[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:42 dist-test-slave-96fg krb5kdc[1232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:42 dist-test-slave-96fg krb5kdc[1232](info): set up 2 sockets
Sep 27 03:53:42 dist-test-slave-96fg krb5kdc[1232](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 03:53:43.514490 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/test-user.keytab.
W20260927 03:53:43.947575 1225 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.311s user 0.000s sys 0.002s
Sep 27 03:53:43 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
W20260927 03:53:45.366703 1225 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.50.126: real 1.399s user 0.001s sys 0.002s
I20260927 03:53:45.367183 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34791
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:45.446385 1245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:45.446586 1245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:45.446607 1245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:45.447758 1245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:45.447803 1245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:45.447815 1245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:45.447826 1245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:45.447837 1245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:45.449349 1245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:45.449589 1245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:45.449834 1245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:53:45.455231 1245 server_base.cc:1034] running on GCE node
W20260927 03:53:45.455327 1251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.455236 1253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.455248 1250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:45.455758 1245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:45.456020 1245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:45.457159 1245 hybrid_clock.cc:648] HybridClock initialized: now 1790481225457142 us; error 36 us; skew 500 ppm
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:45.459522 1245 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:45.459931 1245 webserver.cc:460] Webserver started at http://127.1.50.126:33043/ using document root <none> and password file <none>
I20260927 03:53:45.460134 1245 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:45.460177 1245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:45.460295 1245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:45.461391 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "3c134f65929b4ce5889170e9c4985f24"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "6e1372f094a947d1d9f239584b20f9ff"
server_key_iv: "1bfea7f95257e13fe0b342bcaedeed6d"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.461719 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "3c134f65929b4ce5889170e9c4985f24"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "6e1372f094a947d1d9f239584b20f9ff"
server_key_iv: "1bfea7f95257e13fe0b342bcaedeed6d"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.463011 1245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:53:45.463675 1260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.463805 1245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:53:45.463879 1245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "3c134f65929b4ce5889170e9c4985f24"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "6e1372f094a947d1d9f239584b20f9ff"
server_key_iv: "1bfea7f95257e13fe0b342bcaedeed6d"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.463940 1245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:45.492242 1245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:45.494211 1245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:45.494352 1245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:45.497946 1245 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34791
I20260927 03:53:45.498001 1312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34791 every 8 connection(s)
I20260927 03:53:45.498342 1245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:45.499188 1313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:45.501497 1313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24: Bootstrap starting.
I20260927 03:53:45.502049 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1245
I20260927 03:53:45.502170 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:45.502170 1313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:45.502468 1313 log.cc:824] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:45.502910 1225 external_mini_cluster.cc:1428] Setting key 443958dabe836dfbf3d81372610ad3d5
I20260927 03:53:45.503239 1313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24: No bootstrap required, opened a new log
I20260927 03:53:45.505304 1313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } }
I20260927 03:53:45.505478 1313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:45.505586 1313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c134f65929b4ce5889170e9c4985f24, State: Initialized, Role: FOLLOWER
I20260927 03:53:45.505740 1313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } }
I20260927 03:53:45.505822 1313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:45.505859 1313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:45.505904 1313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:45.507040 1313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } }
I20260927 03:53:45.507149 1313 leader_election.cc:302] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c134f65929b4ce5889170e9c4985f24; no voters:
I20260927 03:53:45.507301 1313 leader_election.cc:288] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:45.507328 1318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:45.507481 1318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [term 1 LEADER]: Becoming Leader. State: Replica: 3c134f65929b4ce5889170e9c4985f24, State: Running, Role: LEADER
I20260927 03:53:45.507534 1313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:45.507663 1318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } }
I20260927 03:53:45.507980 1318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c134f65929b4ce5889170e9c4985f24. Latest consensus state: current_term: 1 leader_uuid: "3c134f65929b4ce5889170e9c4985f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } } }
I20260927 03:53:45.508049 1318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:45.508189 1320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c134f65929b4ce5889170e9c4985f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c134f65929b4ce5889170e9c4985f24" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34791 } } }
I20260927 03:53:45.508251 1320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:45.508379 1325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:45.508999 1325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:45.510576 1325 catalog_manager.cc:1347] Generated new cluster ID: 13650cb1c9d94e9f96dc61da214b6483
I20260927 03:53:45.510629 1325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:45.519927 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.503800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.1.50.126:34791)
0927 03:53:45.505603 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:45.505613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:45.505643 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:53:45.506411 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:45.506416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:45.506458 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:45.506561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:45.508801 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.510004 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.511469 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.512086 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.512217 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.516064 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:45.516068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:45.516069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:45.516081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:45.518011 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.518483 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.518487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.518487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.518595 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.518831 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.518832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.518832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.519148 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:45.519304 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:45.519546 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 03:53:45.519685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1753,"thread_start_us":91,"threads_started":1}
I20260927 03:53:45.523699 1325 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:45.524406 1325 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:45.528298 1325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c134f65929b4ce5889170e9c4985f24: Generated new TSK 0
I20260927 03:53:45.528546 1325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:45.553769 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:45.633857 1339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:45.634116 1339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:45.634145 1339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:45.635342 1339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:45.635385 1339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:45.635413 1339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:53:45.636895 1339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:45.637130 1339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:45.637358 1339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:45.642249 1347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:45.642385 1339 server_base.cc:1034] running on GCE node
W20260927 03:53:45.642387 1344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.642257 1345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:45.642710 1339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:45.642928 1339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:45.644071 1339 hybrid_clock.cc:648] HybridClock initialized: now 1790481225644055 us; error 30 us; skew 500 ppm
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:45.646735 1339 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:53:45.647169 1339 webserver.cc:460] Webserver started at http://127.1.50.65:44583/ using document root <none> and password file <none>
I20260927 03:53:45.647397 1339 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:45.647445 1339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:45.647574 1339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:45.648597 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "47590bf3ea3f472fb527dd75330492ee"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "01390a582f04af26caaf6b1f030630d9"
server_key_iv: "40cb42fbcfae00a2a90612b46f80f1b3"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.648927 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "47590bf3ea3f472fb527dd75330492ee"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "01390a582f04af26caaf6b1f030630d9"
server_key_iv: "40cb42fbcfae00a2a90612b46f80f1b3"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.650216 1339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:53:45.650923 1354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.651073 1339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:45.651156 1339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "47590bf3ea3f472fb527dd75330492ee"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "01390a582f04af26caaf6b1f030630d9"
server_key_iv: "40cb42fbcfae00a2a90612b46f80f1b3"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.651217 1339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:45.662683 1339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:45.664848 1339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:45.664988 1339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:45.665205 1339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:45.665493 1339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:45.665525 1339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.665553 1339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:45.665577 1339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.671880 1339 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:46591
I20260927 03:53:45.671939 1467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:46591 every 8 connection(s)
I20260927 03:53:45.672238 1339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:45.678188 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1339
I20260927 03:53:45.678277 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:53:45.678427 1225 external_mini_cluster.cc:1428] Setting key 2b132072052e850ce0854135292c1af3
I20260927 03:53:45.681495 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.672913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:46593 (local address 127.1.50.126:34791)
0927 03:53:45.672955 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:45.672956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:45.673157 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:53:45.673602 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:45.673603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:45.673615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:45.673688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:45.674626 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.675139 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.676195 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.676398 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.676451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.678639 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:45.678640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:45.678641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:45.678647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:45.680020 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.680410 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.680411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.680412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.680476 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.680851 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.680853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.680853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.680976 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:45.681041 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:45.681359 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 03:53:45.681430 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:53:45.681762 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.672885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34791 (local address 127.1.50.65:46593)
0927 03:53:45.673039 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:45.673075 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:45.673530 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:45.673754 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:45.673758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:45.674359 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:45.674551 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.674556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.675218 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.675219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.676100 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.676103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.676467 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.676467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.676565 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.677062 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:45.677070 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:45.678490 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:45.680119 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.680126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.680128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.680341 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:45.680540 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.680541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.680541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.680755 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:45.681043 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:45.681048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:45.681212 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 03:53:45.681349 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 03:53:45.681516 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260927 03:53:45.682250 1468 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34791
I20260927 03:53:45.682346 1468 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:45.682516 1468 heartbeater.cc:511] Master 127.1.50.126:34791 requested a full tablet report, sending...
I20260927 03:53:45.683145 1277 ts_manager.cc:194] Registered new tserver with Master: 47590bf3ea3f472fb527dd75330492ee (127.1.50.65:46591)
I20260927 03:53:45.684078 1277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:46593
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:53:45.705237 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:45.787801 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:45.788024 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:45.788048 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:45.789196 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:45.789237 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:45.789263 1473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:53:45.790772 1473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:45.790982 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:45.791178 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:45.795926 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.795954 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.795954 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:45.796175 1473 server_base.cc:1034] running on GCE node
I20260927 03:53:45.796382 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:45.796609 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:45.797784 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790481225797764 us; error 30 us; skew 500 ppm
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:45.800665 1473 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:53:45.801175 1473 webserver.cc:460] Webserver started at http://127.1.50.66:44787/ using document root <none> and password file <none>
I20260927 03:53:45.801412 1473 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:45.801465 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:45.801609 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:45.802719 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "6871ef18973340238ce5bfa46135d09d"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "496f115c9ff2668a2c68e0f12b31286f"
server_key_iv: "4a7a105413797951fd85a1e2cbe3a175"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.803074 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "6871ef18973340238ce5bfa46135d09d"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "496f115c9ff2668a2c68e0f12b31286f"
server_key_iv: "4a7a105413797951fd85a1e2cbe3a175"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.804417 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:53:45.805064 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.805213 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:45.805294 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "6871ef18973340238ce5bfa46135d09d"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "496f115c9ff2668a2c68e0f12b31286f"
server_key_iv: "4a7a105413797951fd85a1e2cbe3a175"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.805351 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:45.818423 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:45.820641 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:45.820794 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:45.821007 1473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:45.821321 1473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:45.821354 1473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.821382 1473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:45.821405 1473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.827581 1473 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:44853
I20260927 03:53:45.827661 1601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:44853 every 8 connection(s)
I20260927 03:53:45.827947 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:53:45.829989 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1473
I20260927 03:53:45.830093 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:53:45.830262 1225 external_mini_cluster.cc:1428] Setting key 63453b76b5d84ca00642cadb011b0245
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:45.837347 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.828632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:37487 (local address 127.1.50.126:34791)
0927 03:53:45.828679 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:45.828681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:45.828830 (+ 149us) server_negotiation.cc:407] Connection header received
0927 03:53:45.829318 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:45.829319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:45.829329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:45.829387 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:45.830421 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.830984 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.832305 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.832595 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.832668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.834851 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:45.834852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:45.834853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:45.834860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:45.836125 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.836481 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.836482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.836483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.836532 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.836745 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.836746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.836747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.836848 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:45.836903 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:45.837173 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 03:53:45.837256 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:53:45.837517 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.828570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34791 (local address 127.1.50.66:37487)
0927 03:53:45.828724 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:45.828766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:45.829237 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:45.829461 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:45.829465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:45.830077 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:45.830312 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.830319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.831075 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.831076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.832221 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.832224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.832668 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.832669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.832784 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.833277 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:45.833286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:45.834764 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:45.836197 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.836203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.836206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.836417 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:45.836583 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.836583 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.836584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.836690 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:45.836899 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:45.836905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:45.837040 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:53:45.837143 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:53:45.837300 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260927 03:53:45.837954 1602 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34791
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:53:45.838085 1602 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:45.838274 1602 heartbeater.cc:511] Master 127.1.50.126:34791 requested a full tablet report, sending...
I20260927 03:53:45.838757 1277 ts_manager.cc:194] Registered new tserver with Master: 6871ef18973340238ce5bfa46135d09d (127.1.50.66:44853)
I20260927 03:53:45.839215 1277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:37487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:53:45.857894 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:45.939566 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:45.939779 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:45.939810 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:45.940972 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:45.941018 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:45.941063 1607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:53:45.942571 1607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:45.942802 1607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:45.943003 1607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:53:45.948305 1607 server_base.cc:1034] running on GCE node
W20260927 03:53:45.948266 1615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.948272 1612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:45.948242 1613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:45.948666 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:45.948904 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:45.950089 1607 hybrid_clock.cc:648] HybridClock initialized: now 1790481225950070 us; error 35 us; skew 500 ppm
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:45.952672 1607 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:53:45.953202 1607 webserver.cc:460] Webserver started at http://127.1.50.67:42279/ using document root <none> and password file <none>
I20260927 03:53:45.953418 1607 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:45.953466 1607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:45.953596 1607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:45.954730 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "b3c0e98c2c554017b6fc72e09f587623"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "bcd5900c4f1757d64690391587136fd2"
server_key_iv: "7f100219a75590307aaa2beab199a48c"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.955078 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "b3c0e98c2c554017b6fc72e09f587623"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "bcd5900c4f1757d64690391587136fd2"
server_key_iv: "7f100219a75590307aaa2beab199a48c"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.956365 1607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:53:45.957115 1622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.957312 1607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:45.957396 1607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "b3c0e98c2c554017b6fc72e09f587623"
format_stamp: "Formatted at 2026-09-27 03:53:45 on dist-test-slave-96fg"
server_key: "bcd5900c4f1757d64690391587136fd2"
server_key_iv: "7f100219a75590307aaa2beab199a48c"
server_key_version: "encryptionkey@0"
I20260927 03:53:45.957463 1607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:45.981594 1607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:45.983879 1607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:45.984036 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:45.984284 1607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:45.984587 1607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:45.984622 1607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.984651 1607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:45.984668 1607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:45.990826 1607 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:46503
I20260927 03:53:45.990885 1735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:46503 every 8 connection(s)
I20260927 03:53:45.991192 1607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:53:45.993217 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1607
I20260927 03:53:45.993314 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:53:45.993475 1225 external_mini_cluster.cc:1428] Setting key 96ffba26653d7dfc6cba133fad3945f8
Sep 27 03:53:45 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481225, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:46.000530 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.991893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:56135 (local address 127.1.50.126:34791)
0927 03:53:45.991937 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:45.991938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:45.992141 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:53:45.992675 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:45.992676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:45.992686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:45.992738 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:45.993876 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.994450 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.995418 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.995626 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.995682 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.997806 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:45.997808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:45.997808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:45.997815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:45.999038 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.999436 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.999438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.999438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.999496 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:45.999715 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:45.999716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:45.999716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:45.999857 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:45.999917 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:46.000386 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 03:53:46.000461 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:53:46.000710 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:45.991885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34791 (local address 127.1.50.67:56135)
0927 03:53:45.992022 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:45.992057 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:45.992540 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:45.992794 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:45.992797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:45.993513 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:45.993787 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.993792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.994537 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.994538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.995340 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:45.995342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:45.995695 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:45.995696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:45.995815 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:45.996285 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:45.996294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:45.997722 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:45.999123 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.999129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.999130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.999357 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:45.999555 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:45.999555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:45.999556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:45.999648 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:46.000016 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:46.000022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:46.000251 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 03:53:46.000352 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:53:46.000492 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260927 03:53:46.001178 1736 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34791
I20260927 03:53:46.001298 1736 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:46.001502 1736 heartbeater.cc:511] Master 127.1.50.126:34791 requested a full tablet report, sending...
I20260927 03:53:46.002260 1277 ts_manager.cc:194] Registered new tserver with Master: b3c0e98c2c554017b6fc72e09f587623 (127.1.50.67:46503)
I20260927 03:53:46.002640 1277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:56135
I20260927 03:53:46.005496 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:46.028774 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:46.021223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.1.50.65:46591)
0927 03:53:46.021429 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:46.021436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:46.021477 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:53:46.021510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:46.021513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:46.021532 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:46.021607 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:46.022268 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:46.022987 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:46.023755 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:46.024070 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:46.024156 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:46.026227 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:46.026234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:46.026235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:46.026242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:46.027689 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:46.028006 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:46.028010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:46.028010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:46.028076 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:46.028265 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:46.028266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:46.028266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:46.028394 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:46.028536 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:46.028582 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 03:53:46.028661 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":107,"threads_started":1}
W20260927 03:53:46.029145 1382 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48686
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:46.050575 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:46.043503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.1.50.65:46591)
0927 03:53:46.043613 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:46.043615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:46.043657 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:53:46.043720 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:46.043721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:46.043732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:46.043779 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:46.044318 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:46.044823 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:46.045545 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:46.045789 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:46.045850 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:46.048036 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:46.048038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:46.048038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:46.048048 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:46.049353 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:46.049726 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:46.049727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:46.049728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:46.049787 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:46.049969 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:46.049970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:46.049971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:46.050150 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:46.050266 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:46.050375 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:53:46.050462 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:53:46.051229 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1339
I20260927 03:53:46.055289 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1473
I20260927 03:53:46.059864 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1607
I20260927 03:53:46.063687 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1245
2026-09-27T03:53:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3522 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1750](info): set up 2 sockets
Sep 27 03:53:46 dist-test-slave-96fg krb5kdc[1750](info): commencing operation
krb5kdc: starting...
W20260927 03:53:47.017402 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:47.138237 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38691
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:47.218127 1763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:47.218341 1763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:47.218364 1763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:47.219524 1763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:47.219566 1763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:47.219578 1763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:47.219589 1763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:47.219601 1763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:47.221056 1763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:38691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:47.221305 1763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:47.221515 1763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:47.226177 1768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:47.226264 1763 server_base.cc:1034] running on GCE node
W20260927 03:53:47.226173 1771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.226166 1769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:47.226544 1763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:47.226856 1763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:47.228018 1763 hybrid_clock.cc:648] HybridClock initialized: now 1790481227228005 us; error 28 us; skew 500 ppm
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:47.230501 1763 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:47.230978 1763 webserver.cc:460] Webserver started at http://127.1.50.126:44469/ using document root <none> and password file <none>
I20260927 03:53:47.231212 1763 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:47.231261 1763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:47.231381 1763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:47.232461 1763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "52d1e15e4e807eb19372ef0c02f667a8"
server_key_iv: "116c216bf3eecc4c136ece2c36bb03c0"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.232796 1763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "52d1e15e4e807eb19372ef0c02f667a8"
server_key_iv: "116c216bf3eecc4c136ece2c36bb03c0"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.234066 1763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:53:47.234795 1778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.234942 1763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:47.235024 1763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "52d1e15e4e807eb19372ef0c02f667a8"
server_key_iv: "116c216bf3eecc4c136ece2c36bb03c0"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.235100 1763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:47.249130 1763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:47.251289 1763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:47.251441 1763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:47.255262 1763 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:38691
I20260927 03:53:47.255342 1830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:38691 every 8 connection(s)
I20260927 03:53:47.255640 1763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:47.256511 1831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:47.258767 1831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e: Bootstrap starting.
I20260927 03:53:47.259285 1831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:47.259510 1831 log.cc:824] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:47.260138 1831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e: No bootstrap required, opened a new log
I20260927 03:53:47.261776 1831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } }
I20260927 03:53:47.261916 1831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:47.261937 1831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4a67ffc17e4954a6e4b6119fd8a26e, State: Initialized, Role: FOLLOWER
I20260927 03:53:47.262080 1831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } }
I20260927 03:53:47.262149 1831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:47.262171 1831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:47.262202 1831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:47.262676 1831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } }
I20260927 03:53:47.262768 1831 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c4a67ffc17e4954a6e4b6119fd8a26e; no voters:
I20260927 03:53:47.262904 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1763
I20260927 03:53:47.262929 1831 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:47.262985 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:47.262981 1834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:47.263139 1225 external_mini_cluster.cc:1428] Setting key 78fbcb7464aa549bb958c52628dc4d82
I20260927 03:53:47.263168 1831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:47.263280 1834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [term 1 LEADER]: Becoming Leader. State: Replica: 0c4a67ffc17e4954a6e4b6119fd8a26e, State: Running, Role: LEADER
I20260927 03:53:47.263365 1834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } }
I20260927 03:53:47.263705 1834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4a67ffc17e4954a6e4b6119fd8a26e. Latest consensus state: current_term: 1 leader_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } } }
I20260927 03:53:47.263744 1836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a67ffc17e4954a6e4b6119fd8a26e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38691 } } }
I20260927 03:53:47.263873 1834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:47.263944 1836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:47.264119 1845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:47.264654 1845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:47.266407 1845 catalog_manager.cc:1347] Generated new cluster ID: b84e4696c94246679e0913d751f98666
I20260927 03:53:47.266455 1845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:47.273171 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.263611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.1.50.126:38691)
0927 03:53:47.263998 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.264009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.264045 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:53:47.264180 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.264188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.264257 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.264402 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.265126 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.266401 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.267180 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.267600 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.267690 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.269778 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.269782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.269782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.269793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.271731 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.272070 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.272073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.272073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.272169 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.272343 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.272344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.272345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.272577 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.272737 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.272804 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.272936 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
I20260927 03:53:47.281893 1845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:47.282534 1845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:47.286405 1845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c4a67ffc17e4954a6e4b6119fd8a26e: Generated new TSK 0
I20260927 03:53:47.286630 1845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:47.314787 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:47.396450 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:47.396658 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:47.396689 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:47.397832 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:47.397871 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:47.397904 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:53:47.399426 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:47.399657 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:47.399856 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:47.404714 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.404759 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.404719 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:47.404924 1857 server_base.cc:1034] running on GCE node
I20260927 03:53:47.405179 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:47.405396 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:47.406597 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790481227406523 us; error 38 us; skew 500 ppm
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:47.409073 1857 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:53:47.409548 1857 webserver.cc:460] Webserver started at http://127.1.50.65:34071/ using document root <none> and password file <none>
I20260927 03:53:47.409767 1857 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:47.409818 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:47.409942 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:47.411005 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "b34681dcb97147e1be3e07800bc94d15"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "184b55cfaa6eac991b2ba3eec8580150"
server_key_iv: "1107be78a983ad2a23e51c7907caa0bc"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.411338 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "b34681dcb97147e1be3e07800bc94d15"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "184b55cfaa6eac991b2ba3eec8580150"
server_key_iv: "1107be78a983ad2a23e51c7907caa0bc"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.412611 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:53:47.413391 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.413552 1857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:47.413659 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "b34681dcb97147e1be3e07800bc94d15"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "184b55cfaa6eac991b2ba3eec8580150"
server_key_iv: "1107be78a983ad2a23e51c7907caa0bc"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.413725 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:47.423231 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:47.425139 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:47.425274 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:47.425488 1857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:47.425789 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:47.425822 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.425850 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:47.425868 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.431867 1857 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:39321
I20260927 03:53:47.432178 1985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:39321 every 8 connection(s)
I20260927 03:53:47.432400 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:47.439572 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1857
I20260927 03:53:47.439693 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:53:47.439813 1225 external_mini_cluster.cc:1428] Setting key 32617fe5804486b3310189c4e2722b7a
I20260927 03:53:47.442111 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.433309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:42543 (local address 127.1.50.126:38691)
0927 03:53:47.433364 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.433366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.433562 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:53:47.434688 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.434689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.434706 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.434757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.435413 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.435916 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.436919 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.437135 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.437194 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.439357 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.439358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.439359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.439366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.440804 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.441200 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.441201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.441202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.441266 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.441485 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.441486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.441487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.441600 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.441655 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.441913 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.442000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:53:47.442286 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.433237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38691 (local address 127.1.50.65:42543)
0927 03:53:47.433430 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.433474 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.434567 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.434832 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.434836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.435113 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:47.435316 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.435320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.435999 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.436000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.436831 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.436833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.437211 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.437212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.437332 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.437809 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:47.437818 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:47.439265 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:47.440880 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.440887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.440890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.441130 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.441319 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.441320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.441320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.441428 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.441653 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:47.441659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:47.441799 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.441886 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.442060 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260927 03:53:47.442719 1986 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38691
I20260927 03:53:47.442808 1986 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:47.442977 1986 heartbeater.cc:511] Master 127.1.50.126:38691 requested a full tablet report, sending...
I20260927 03:53:47.443502 1795 ts_manager.cc:194] Registered new tserver with Master: b34681dcb97147e1be3e07800bc94d15 (127.1.50.65:39321)
I20260927 03:53:47.444397 1795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:42543
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:53:47.465418 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:47.544612 1991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:47.544828 1991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:47.544853 1991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:47.546042 1991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:47.546090 1991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:47.546119 1991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:53:47.547621 1991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:47.547833 1991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:47.548053 1991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:47.552937 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.552973 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.552950 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:47.552989 1991 server_base.cc:1034] running on GCE node
I20260927 03:53:47.553363 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:47.553588 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:47.554726 1991 hybrid_clock.cc:648] HybridClock initialized: now 1790481227554703 us; error 36 us; skew 500 ppm
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:47.557188 1991 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:53:47.557615 1991 webserver.cc:460] Webserver started at http://127.1.50.66:33969/ using document root <none> and password file <none>
I20260927 03:53:47.557821 1991 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:47.557868 1991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:47.557999 1991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:47.559051 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "24e9588a732a4abdbb2b66e3a9d515cc"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "441d64b739cc0c77ff2009adfa57d61e"
server_key_iv: "46c81ad174113ec1d9c42ae003bc1c07"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.559367 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "24e9588a732a4abdbb2b66e3a9d515cc"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "441d64b739cc0c77ff2009adfa57d61e"
server_key_iv: "46c81ad174113ec1d9c42ae003bc1c07"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.560616 1991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:53:47.561388 2006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.561542 1991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:47.561635 1991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "24e9588a732a4abdbb2b66e3a9d515cc"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "441d64b739cc0c77ff2009adfa57d61e"
server_key_iv: "46c81ad174113ec1d9c42ae003bc1c07"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.561699 1991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:47.584672 1991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:47.586472 1991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:47.586611 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:47.586824 1991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:47.587110 1991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:47.587143 1991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.587172 1991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:47.587195 1991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.592629 1991 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:44383
I20260927 03:53:47.592682 2119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:44383 every 8 connection(s)
I20260927 03:53:47.592993 1991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:47.600210 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 1991
I20260927 03:53:47.600286 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:53:47.600430 1225 external_mini_cluster.cc:1428] Setting key 6e374e9d13e6265dd50a2387d07dfc34
I20260927 03:53:47.603005 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.593775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:47363 (local address 127.1.50.126:38691)
0927 03:53:47.593871 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.593873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.594035 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:53:47.595087 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.595088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.595102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.595167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.595831 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.596341 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.597379 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.597592 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.597658 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.599969 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.599970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.599971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.599979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.601534 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.601985 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.601986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.601987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.602123 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.602359 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.602360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.602361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.602481 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.602546 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.602809 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.602903 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 03:53:47.603174 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.593666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38691 (local address 127.1.50.66:47363)
0927 03:53:47.593876 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.593926 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.594950 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.595241 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.595245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.595535 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:47.595731 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.595736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.596440 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.596441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.597287 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.597289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.597668 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.597668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.597803 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.598348 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:47.598357 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:47.599867 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:47.601656 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.601663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.601666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.601911 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.602179 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.602181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.602181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.602299 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.602541 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:47.602547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:47.602689 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.602785 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.602948 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260927 03:53:47.603677 2120 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38691
I20260927 03:53:47.603780 2120 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:47.603979 2120 heartbeater.cc:511] Master 127.1.50.126:38691 requested a full tablet report, sending...
I20260927 03:53:47.604408 1795 ts_manager.cc:194] Registered new tserver with Master: 24e9588a732a4abdbb2b66e3a9d515cc (127.1.50.66:44383)
I20260927 03:53:47.604882 1795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:47363
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:53:47.625329 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:47.707612 2125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:47.707819 2125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:47.707842 2125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:47.708948 2125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:47.708984 2125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:47.709010 2125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:53:47.710470 2125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:47.710675 2125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:47.710860 2125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:47.715791 2133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.715868 2131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:47.715792 2130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:47.715996 2125 server_base.cc:1034] running on GCE node
I20260927 03:53:47.716262 2125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:47.716490 2125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:47.717667 2125 hybrid_clock.cc:648] HybridClock initialized: now 1790481227717645 us; error 33 us; skew 500 ppm
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:47.720301 2125 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:53:47.720739 2125 webserver.cc:460] Webserver started at http://127.1.50.67:38767/ using document root <none> and password file <none>
I20260927 03:53:47.720950 2125 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:47.720995 2125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:47.721105 2125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:47.722179 2125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "f14c4474fe3a46e2b3f695f8cde3c757"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "e99f43bb04831cb4027bfa2dbf78aff9"
server_key_iv: "59918fbe1347cc6ee3ae81eac1736cd9"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.722482 2125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "f14c4474fe3a46e2b3f695f8cde3c757"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "e99f43bb04831cb4027bfa2dbf78aff9"
server_key_iv: "59918fbe1347cc6ee3ae81eac1736cd9"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.723784 2125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:47.724581 2140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.724786 2125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:47.724875 2125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "f14c4474fe3a46e2b3f695f8cde3c757"
format_stamp: "Formatted at 2026-09-27 03:53:47 on dist-test-slave-96fg"
server_key: "e99f43bb04831cb4027bfa2dbf78aff9"
server_key_iv: "59918fbe1347cc6ee3ae81eac1736cd9"
server_key_version: "encryptionkey@0"
I20260927 03:53:47.724936 2125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:47.732652 2125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:47.734638 2125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:47.734773 2125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:47.734972 2125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:47.735252 2125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:47.735284 2125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.735312 2125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:47.735337 2125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:47.740983 2125 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42299
I20260927 03:53:47.741078 2253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42299 every 8 connection(s)
I20260927 03:53:47.741365 2125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:47.749948 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2125
I20260927 03:53:47.750059 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:53:47.750226 1225 external_mini_cluster.cc:1428] Setting key c3b569912ea9369e2851d007955285d3
I20260927 03:53:47.750276 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.742151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:58977 (local address 127.1.50.126:38691)
0927 03:53:47.742222 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.742223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.742373 (+ 150us) server_negotiation.cc:407] Connection header received
0927 03:53:47.743287 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.743288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.743298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.743353 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.743927 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.744406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.745491 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.745680 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.745741 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.747757 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.747758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.747759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.747766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.748965 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.749370 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.749371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.749371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.749436 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.749634 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.749635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.749635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.749741 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.749789 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.750082 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.750161 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:53:47.750417 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.742100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38691 (local address 127.1.50.67:58977)
0927 03:53:47.742254 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.742295 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.743193 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.743418 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.743421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.743680 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:47.743867 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.743871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.744549 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.744551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.745396 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.745398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.745769 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.745771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.745876 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.746364 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:47.746373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:47.747665 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:47.749046 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.749052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.749054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.749288 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.749495 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:47.749496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:47.749496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:47.749586 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:47.749802 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:47.749805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:47.749930 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.750013 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.750205 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260927 03:53:47.750890 2254 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38691
I20260927 03:53:47.750979 2254 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:47.751122 2254 heartbeater.cc:511] Master 127.1.50.126:38691 requested a full tablet report, sending...
I20260927 03:53:47.751446 1794 ts_manager.cc:194] Registered new tserver with Master: f14c4474fe3a46e2b3f695f8cde3c757 (127.1.50.67:42299)
I20260927 03:53:47.751822 1794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:58977
I20260927 03:53:47.762506 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:53:47.770342 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.764237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.1.50.126:38691)
0927 03:53:47.764279 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.764281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.764423 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:53:47.764490 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.764491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.764502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.764551 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.765210 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.765769 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.766572 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.766765 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.766820 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.767954 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.767955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.767955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.767961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.769283 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.769615 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.769616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.769616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.769674 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.769817 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.769817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.769817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.769952 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.770062 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.770202 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.770265 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:53:47.772980 1794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:47.774556 1794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:47.780467 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.1.50.65:39321)
0927 03:53:47.777172 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.777176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.777184 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:47.777215 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.777218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.777231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.777300 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:47.777477 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778534 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.779467 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780103 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780165 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780223 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.780351 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.780389 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260927 03:53:47.780460 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39321 (local address 127.0.0.1:48450)
0927 03:53:47.776981 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.776989 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.777022 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.777373 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.777374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.777377 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:47.777408 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.777409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778631 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.778632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.779386 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.779388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780179 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.780255 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780269 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.780332 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.780385 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":35,"threads_started":1}
I20260927 03:53:47.780721 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:44383 (local address 127.0.0.1:39566)
0927 03:53:47.776784 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.776823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.776880 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.777164 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.777167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.777170 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:47.777237 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.777239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778315 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.778315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.779512 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.779514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780480 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.780549 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780556 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.780599 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.780626 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":30,"thread_start_us":54,"threads_started":1}
I20260927 03:53:47.780768 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.1.50.66:44383)
0927 03:53:47.776885 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.776889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.776900 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:53:47.776967 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.776970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.776985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.777067 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:47.777317 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778235 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.779621 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780335 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780400 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780453 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.780662 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.780700 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260927 03:53:47.780720 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42299 (local address 127.0.0.1:45772)
0927 03:53:47.776967 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.776976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.777017 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.777253 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.777253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.777255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:47.777309 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.777311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778576 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.778577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.779647 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.779649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780469 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.780549 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780555 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.780583 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.780622 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260927 03:53:47.781070 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.776708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.1.50.67:42299)
0927 03:53:47.777125 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.777130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.777138 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:47.777169 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.777171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.777182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.777249 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:47.777361 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.778473 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.779725 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.780386 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.780803 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.780908 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.780937 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.780996 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":88,"threads_started":1}
I20260927 03:53:47.781563 2054 tablet_service.cc:1467] Processing CreateTablet for tablet d067abe98b174364baae20e0d6f6937d (DEFAULT_TABLE table=test-table [id=e426d2130f074f8ab3b81fbaacb30376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:47.781571 1920 tablet_service.cc:1467] Processing CreateTablet for tablet d067abe98b174364baae20e0d6f6937d (DEFAULT_TABLE table=test-table [id=e426d2130f074f8ab3b81fbaacb30376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:47.781632 2188 tablet_service.cc:1467] Processing CreateTablet for tablet d067abe98b174364baae20e0d6f6937d (DEFAULT_TABLE table=test-table [id=e426d2130f074f8ab3b81fbaacb30376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:47.781920 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d067abe98b174364baae20e0d6f6937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:47.781922 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d067abe98b174364baae20e0d6f6937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:47.781941 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d067abe98b174364baae20e0d6f6937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:47.784549 2268 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: Bootstrap starting.
I20260927 03:53:47.785048 2270 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Bootstrap starting.
I20260927 03:53:47.785224 2268 tablet_bootstrap.cc:654] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:47.785478 2269 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Bootstrap starting.
I20260927 03:53:47.785656 2270 tablet_bootstrap.cc:654] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:47.785743 2268 log.cc:824] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:47.786085 2269 tablet_bootstrap.cc:654] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:47.786243 2270 log.cc:824] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:47.786546 2268 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: No bootstrap required, opened a new log
I20260927 03:53:47.786605 2268 ts_tablet_manager.cc:1397] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:53:47.786613 2269 log.cc:824] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:47.786926 2270 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: No bootstrap required, opened a new log
I20260927 03:53:47.786993 2270 ts_tablet_manager.cc:1397] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:47.787474 2269 tablet_bootstrap.cc:492] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: No bootstrap required, opened a new log
I20260927 03:53:47.787541 2269 ts_tablet_manager.cc:1397] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:53:47.788187 2268 raft_consensus.cc:348] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.788365 2268 raft_consensus.cc:374] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:47.788425 2268 raft_consensus.cc:729] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14c4474fe3a46e2b3f695f8cde3c757, State: Initialized, Role: FOLLOWER
I20260927 03:53:47.788542 2268 consensus_queue.cc:260] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.788789 2268 ts_tablet_manager.cc:1428] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:53:47.788900 2254 heartbeater.cc:503] Master 127.1.50.126:38691 was elected leader, sending a full tablet report...
I20260927 03:53:47.788857 2270 raft_consensus.cc:348] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.789026 2270 raft_consensus.cc:374] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:47.789068 2270 raft_consensus.cc:729] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34681dcb97147e1be3e07800bc94d15, State: Initialized, Role: FOLLOWER
I20260927 03:53:47.789187 2270 consensus_queue.cc:260] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.789453 2270 ts_tablet_manager.cc:1428] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:53:47.789453 1986 heartbeater.cc:503] Master 127.1.50.126:38691 was elected leader, sending a full tablet report...
I20260927 03:53:47.789831 2269 raft_consensus.cc:348] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.789997 2269 raft_consensus.cc:374] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:47.790096 2269 raft_consensus.cc:729] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e9588a732a4abdbb2b66e3a9d515cc, State: Initialized, Role: FOLLOWER
I20260927 03:53:47.790215 2269 consensus_queue.cc:260] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.790468 2269 ts_tablet_manager.cc:1428] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:53:47.790468 2120 heartbeater.cc:503] Master 127.1.50.126:38691 was elected leader, sending a full tablet report...
W20260927 03:53:47.843532 2121 tablet.cc:2369] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:53:47.903816 2276 raft_consensus.cc:482] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:47.903970 2276 raft_consensus.cc:504] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.904398 2276 leader_election.cc:288] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14c4474fe3a46e2b3f695f8cde3c757 (127.1.50.67:42299), b34681dcb97147e1be3e07800bc94d15 (127.1.50.65:39321)
I20260927 03:53:47.908236 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.904577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39321 (local address 127.1.50.66:45021)
0927 03:53:47.904681 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.904693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.904744 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.904920 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.904921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.904922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:47.904987 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.904989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.905914 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.905914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.907165 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.907168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.907956 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.908046 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.908048 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.908105 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.908165 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":84}
I20260927 03:53:47.908259 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.904687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:45021 (local address 127.1.50.65:39321)
0927 03:53:47.904726 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.904727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.904737 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:53:47.904831 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.904832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.904844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.904905 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:47.905056 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.905837 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907265 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.907871 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907934 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.907943 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.908143 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.908186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:53:47.908259 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.904699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:60923 (local address 127.1.50.67:42299)
0927 03:53:47.904745 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.904746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.904871 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:53:47.904909 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.904911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.904922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.904970 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:47.905107 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.905787 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907265 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.907846 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907925 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.907937 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.908143 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.908186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:53:47.908242 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.904648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42299 (local address 127.1.50.66:60923)
0927 03:53:47.904801 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:47.904810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:47.904854 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:47.904980 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:47.904981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:47.904982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:47.905031 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.905032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.905881 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.905883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.907179 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:47.907180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.907956 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.907957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:47.908043 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.908048 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:53:47.908105 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:53:47.908148 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260927 03:53:47.908596 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d067abe98b174364baae20e0d6f6937d" candidate_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34681dcb97147e1be3e07800bc94d15" is_pre_election: true
I20260927 03:53:47.908635 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d067abe98b174364baae20e0d6f6937d" candidate_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" is_pre_election: true
I20260927 03:53:47.908751 1940 raft_consensus.cc:2454] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e9588a732a4abdbb2b66e3a9d515cc in term 0.
I20260927 03:53:47.908777 2208 raft_consensus.cc:2454] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e9588a732a4abdbb2b66e3a9d515cc in term 0.
I20260927 03:53:47.908950 2007 leader_election.cc:302] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e9588a732a4abdbb2b66e3a9d515cc, b34681dcb97147e1be3e07800bc94d15; no voters:
I20260927 03:53:47.909125 2276 raft_consensus.cc:2781] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:47.909163 2276 raft_consensus.cc:482] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:47.909185 2276 raft_consensus.cc:3037] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:47.909828 2276 raft_consensus.cc:504] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.909947 2276 leader_election.cc:288] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [CANDIDATE]: Term 1 election: Requested vote from peers f14c4474fe3a46e2b3f695f8cde3c757 (127.1.50.67:42299), b34681dcb97147e1be3e07800bc94d15 (127.1.50.65:39321)
I20260927 03:53:47.910120 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d067abe98b174364baae20e0d6f6937d" candidate_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34681dcb97147e1be3e07800bc94d15"
I20260927 03:53:47.910140 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d067abe98b174364baae20e0d6f6937d" candidate_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14c4474fe3a46e2b3f695f8cde3c757"
I20260927 03:53:47.910202 1940 raft_consensus.cc:3037] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:47.910209 2208 raft_consensus.cc:3037] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:47.910840 2208 raft_consensus.cc:2454] T d067abe98b174364baae20e0d6f6937d P f14c4474fe3a46e2b3f695f8cde3c757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e9588a732a4abdbb2b66e3a9d515cc in term 1.
I20260927 03:53:47.910909 1940 raft_consensus.cc:2454] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e9588a732a4abdbb2b66e3a9d515cc in term 1.
I20260927 03:53:47.911015 2008 leader_election.cc:302] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e9588a732a4abdbb2b66e3a9d515cc, f14c4474fe3a46e2b3f695f8cde3c757; no voters:
I20260927 03:53:47.911109 2276 raft_consensus.cc:2781] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:47.911266 2276 raft_consensus.cc:686] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [term 1 LEADER]: Becoming Leader. State: Replica: 24e9588a732a4abdbb2b66e3a9d515cc, State: Running, Role: LEADER
I20260927 03:53:47.911352 2276 consensus_queue.cc:237] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } }
I20260927 03:53:47.912218 1795 catalog_manager.cc:5495] T d067abe98b174364baae20e0d6f6937d P 24e9588a732a4abdbb2b66e3a9d515cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 24e9588a732a4abdbb2b66e3a9d515cc (127.1.50.66). New cstate: current_term: 1 leader_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14c4474fe3a46e2b3f695f8cde3c757" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e9588a732a4abdbb2b66e3a9d515cc" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b34681dcb97147e1be3e07800bc94d15" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39321 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:47.929097 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.921996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.1.50.65:39321)
0927 03:53:47.922112 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.922114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.922186 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:53:47.922289 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.922290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.922300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.922359 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.922854 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.923344 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.924000 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.924308 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.924386 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.926556 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.926565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.926565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.926571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.928099 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.928408 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.928411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.928412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.928470 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.928608 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.928649 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.928650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.928786 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.928896 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.928971 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.929028 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 03:53:47.933197 1987 tablet.cc:2369] T d067abe98b174364baae20e0d6f6937d P b34681dcb97147e1be3e07800bc94d15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:47.950069 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.943483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.1.50.65:39321)
0927 03:53:47.943538 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.943540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.943604 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:53:47.943675 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.943676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.943685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.943735 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.944265 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.944731 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.945426 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.945630 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.945683 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.947838 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.947840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.947840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.947847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.949016 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.949329 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.949330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.949330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.949380 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.949581 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.949581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.949582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.949704 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.949771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.949914 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.949971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:53:47 dist-test-slave-96fg krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:47.971984 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:47.965297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.1.50.65:39321)
0927 03:53:47.965355 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:47.965357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:47.965452 (+ 95us) server_negotiation.cc:407] Connection header received
0927 03:53:47.965518 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:47.965519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:47.965528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:47.965566 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:47.966138 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.966601 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.967306 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:47.967505 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:47.967560 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:47.969673 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:47.969675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:47.969675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:47.969681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:47.971002 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.971332 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.971333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.971333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.971380 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:47.971589 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:47.971590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:47.971590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:47.971679 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:47.971755 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:47.971861 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:53:47.971917 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 03:53:47.972426 1900 tablet_service.cc:2908] Rejecting scan request for tablet d067abe98b174364baae20e0d6f6937d: Uninitialized: clean time has not yet been initialized
I20260927 03:53:47.973255 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1857
I20260927 03:53:47.977355 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1991
I20260927 03:53:47.981446 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2125
I20260927 03:53:47.985152 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 1763
2026-09-27T03:53:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1921 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:48 dist-test-slave-96fg krb5kdc[2292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:48 dist-test-slave-96fg krb5kdc[2292](info): set up 2 sockets
Sep 27 03:53:48 dist-test-slave-96fg krb5kdc[2292](info): commencing operation
krb5kdc: starting...
W20260927 03:53:48.941499 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:49.062279 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38607
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:49.142915 2305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:49.143203 2305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:49.143240 2305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:49.144549 2305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:49.144600 2305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:53:49.144613 2305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:49.144625 2305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:49.144636 2305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:49.144646 2305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:49.146178 2305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:38607
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:49.146405 2305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:49.146612 2305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:49.151364 2313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.151418 2310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.151374 2311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.151765 2305 server_base.cc:1034] running on GCE node
I20260927 03:53:49.152043 2305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:49.152338 2305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:49.153502 2305 hybrid_clock.cc:648] HybridClock initialized: now 1790481229153479 us; error 38 us; skew 500 ppm
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:49.156090 2305 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:49.156543 2305 webserver.cc:460] Webserver started at http://127.1.50.126:33103/ using document root <none> and password file <none>
I20260927 03:53:49.156761 2305 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:49.156831 2305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:49.156952 2305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:49.157982 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "0881f71c5ca84684a8e5a994c3900c32"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "4a4fd85fd2a82758b31f27bd01341b36"
server_key_iv: "51665330329cec0676e69dcf58fc33e6"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.158355 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "0881f71c5ca84684a8e5a994c3900c32"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "4a4fd85fd2a82758b31f27bd01341b36"
server_key_iv: "51665330329cec0676e69dcf58fc33e6"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.159639 2305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.160372 2320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.160544 2305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.160635 2305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "0881f71c5ca84684a8e5a994c3900c32"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "4a4fd85fd2a82758b31f27bd01341b36"
server_key_iv: "51665330329cec0676e69dcf58fc33e6"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.160712 2305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:49.170656 2305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:49.173032 2305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:49.173193 2305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:49.177647 2305 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:38607
I20260927 03:53:49.177706 2382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:38607 every 8 connection(s)
I20260927 03:53:49.178009 2305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:49.178733 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.181016 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Bootstrap starting.
I20260927 03:53:49.181602 2383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.181880 2383 log.cc:824] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:49.182519 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: No bootstrap required, opened a new log
I20260927 03:53:49.184573 2383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } }
I20260927 03:53:49.184716 2383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.184737 2383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0881f71c5ca84684a8e5a994c3900c32, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.184881 2383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } }
I20260927 03:53:49.184952 2383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:49.184973 2383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:49.184999 2383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.185540 2383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } }
I20260927 03:53:49.185624 2383 leader_election.cc:302] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0881f71c5ca84684a8e5a994c3900c32; no voters:
I20260927 03:53:49.185779 2383 leader_election.cc:288] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:49.185835 2386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:49.186005 2383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:49.186039 2386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [term 1 LEADER]: Becoming Leader. State: Replica: 0881f71c5ca84684a8e5a994c3900c32, State: Running, Role: LEADER
I20260927 03:53:49.186197 2386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } }
I20260927 03:53:49.186532 2388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0881f71c5ca84684a8e5a994c3900c32. Latest consensus state: current_term: 1 leader_uuid: "0881f71c5ca84684a8e5a994c3900c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } } }
I20260927 03:53:49.186537 2387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0881f71c5ca84684a8e5a994c3900c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0881f71c5ca84684a8e5a994c3900c32" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38607 } } }
I20260927 03:53:49.186695 2387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:49.186689 2388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:49.186929 2395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:49.187345 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2305
I20260927 03:53:49.187425 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:49.187453 2395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:49.187582 1225 external_mini_cluster.cc:1428] Setting key 6065f275f8820d7299350d972b1e311c
I20260927 03:53:49.189265 2395 catalog_manager.cc:1347] Generated new cluster ID: 0724d75842b94a1a86b731ec6d47aee9
I20260927 03:53:49.189347 2395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.200390 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.188105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.1.50.126:38607)
0927 03:53:49.189363 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.189373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.189406 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:53:49.189525 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.189531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.189586 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.189731 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.190498 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.191907 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.193046 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.193718 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.193826 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.196583 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.196589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.196589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.196604 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.198598 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.199017 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.199022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.199022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.199172 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.199368 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.199369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.199369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.199720 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.199882 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.199954 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.200136 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":1148,"threads_started":1}
I20260927 03:53:49.205960 2395 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:49.206609 2395 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:49.211614 2395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Generated new TSK 0
I20260927 03:53:49.211823 2395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:49.246580 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38607
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:49.328058 2409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:49.328258 2409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:49.328280 2409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:49.329367 2409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:49.329398 2409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:49.329416 2409 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:49.329432 2409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:53:49.330842 2409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:49.331019 2409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:49.331187 2409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:49.335915 2414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.335925 2415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.335959 2409 server_base.cc:1034] running on GCE node
W20260927 03:53:49.335915 2417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.336308 2409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:49.336531 2409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:49.337706 2409 hybrid_clock.cc:648] HybridClock initialized: now 1790481229337693 us; error 24 us; skew 500 ppm
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:49.340296 2409 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:53:49.340780 2409 webserver.cc:460] Webserver started at http://127.1.50.65:39193/ using document root <none> and password file <none>
I20260927 03:53:49.340996 2409 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:49.341043 2409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:49.341173 2409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:49.342258 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "af22044f24c04bb2abd6ede44da4ad07"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1967afa49189be8f5e266a50379955ce"
server_key_iv: "fb6d8bc13ea29f9d18849244a98d58eb"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.342576 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "af22044f24c04bb2abd6ede44da4ad07"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1967afa49189be8f5e266a50379955ce"
server_key_iv: "fb6d8bc13ea29f9d18849244a98d58eb"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.343806 2409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.344514 2424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.344666 2409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.344745 2409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "af22044f24c04bb2abd6ede44da4ad07"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1967afa49189be8f5e266a50379955ce"
server_key_iv: "fb6d8bc13ea29f9d18849244a98d58eb"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.344805 2409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:49.362283 2409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:49.364189 2409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:49.364323 2409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:49.364869 2409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:49.364912 2409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.364940 2409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:49.364956 2409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.371415 2409 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:33227
I20260927 03:53:49.371486 2539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:33227 every 8 connection(s)
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.371809 2409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:53:49.374965 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.365287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:56243 (local address 127.1.50.126:38607)
0927 03:53:49.365339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.365342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.365534 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:53:49.366125 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.366126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.366140 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.366206 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.366848 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.367615 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.369191 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.369515 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.369591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.372222 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.372224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.372224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.372236 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.373509 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.373918 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.373919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.373919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.373970 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.374292 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.374294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.374294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.374417 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.374480 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.374820 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.374900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:53:49.375171 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.365261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.65:56243)
0927 03:53:49.365407 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.365448 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.365980 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.366273 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.366277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.366552 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.366752 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.366757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.367735 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.367737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.369090 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.369093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.369602 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.369603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.369847 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.370476 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.370487 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.372069 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.373579 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.373584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.373586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.373859 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.374082 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.374084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.374084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.374203 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.374530 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.374536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.374673 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.374765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.374967 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260927 03:53:49.375676 2540 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38607
I20260927 03:53:49.375761 2540 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:49.375958 2540 heartbeater.cc:511] Master 127.1.50.126:38607 requested a full tablet report, sending...
I20260927 03:53:49.376502 2336 ts_manager.cc:194] Registered new tserver with Master: af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227)
I20260927 03:53:49.377182 2336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:56243
I20260927 03:53:49.381228 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2409
I20260927 03:53:49.381320 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:53:49.381479 1225 external_mini_cluster.cc:1428] Setting key 334d858ebba394a5740c407a1db37fe4
I20260927 03:53:49.381646 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.376723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35629 (local address 127.1.50.126:38607)
0927 03:53:49.376759 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.376761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.376892 (+ 131us) server_negotiation.cc:407] Connection header received
0927 03:53:49.376940 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.376941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.376950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.377001 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.377293 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.377800 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.378519 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.378722 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.378775 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.379626 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.379627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.379627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.379633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.380686 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.380963 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.380964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.380964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.381015 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.381174 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.381175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.381176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.381290 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.381344 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.381484 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.381552 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:53:49.381654 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.376682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.65:35629)
0927 03:53:49.376827 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.376836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.376888 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.377007 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.377007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.377174 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.377230 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.377232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.377883 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.377884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.378450 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.378452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.378777 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.378778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.378849 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.379216 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.379220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.379491 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.380752 (+ 1261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.380753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.380754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.380907 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.381064 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.381065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.381065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.381126 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.381344 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.381345 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.381388 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.381456 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.381552 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:53:49.405710 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38607
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:49.484916 2549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:49.485132 2549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:49.485155 2549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:49.486384 2549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:49.486433 2549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:49.486454 2549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:49.486470 2549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:53:49.488032 2549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:49.488247 2549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:49.488433 2549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:49.493178 2555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.493206 2554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.493273 2557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.493464 2549 server_base.cc:1034] running on GCE node
I20260927 03:53:49.493681 2549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:49.493897 2549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:49.495044 2549 hybrid_clock.cc:648] HybridClock initialized: now 1790481229495030 us; error 25 us; skew 500 ppm
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:49.497427 2549 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:53:49.497829 2549 webserver.cc:460] Webserver started at http://127.1.50.66:43047/ using document root <none> and password file <none>
I20260927 03:53:49.498039 2549 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:49.498095 2549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:49.498189 2549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:49.499508 2549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "be8abf885dda4495ab9bfe779419f141"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1f7c62e36156ebe95d21d09bc9494aae"
server_key_iv: "268fea3f429a31d9f4ce323817e74784"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.499934 2549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "be8abf885dda4495ab9bfe779419f141"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1f7c62e36156ebe95d21d09bc9494aae"
server_key_iv: "268fea3f429a31d9f4ce323817e74784"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.501289 2549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.502082 2564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.502278 2549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.502377 2549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "be8abf885dda4495ab9bfe779419f141"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "1f7c62e36156ebe95d21d09bc9494aae"
server_key_iv: "268fea3f429a31d9f4ce323817e74784"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.502450 2549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:49.511255 2549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:49.513196 2549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:49.513330 2549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:49.513928 2549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:49.513969 2549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.514002 2549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:49.514056 2549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.520251 2549 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:37927
I20260927 03:53:49.520308 2679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:37927 every 8 connection(s)
I20260927 03:53:49.520646 2549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:53:49.523041 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.514267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:40473 (local address 127.1.50.126:38607)
0927 03:53:49.514311 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.514314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.514548 (+ 234us) server_negotiation.cc:407] Connection header received
0927 03:53:49.515014 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.515016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.515027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.515117 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.515838 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.516579 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.517999 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.518311 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.518384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.520469 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.520470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.520470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.520479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.521803 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.522200 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.522201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.522201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.522253 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.522496 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.522497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.522497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.522608 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.522662 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.522915 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.522977 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:53:49.523203 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.514262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.66:40473)
0927 03:53:49.514420 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.514458 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.514930 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.515170 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.515174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.515461 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.515741 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.515747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.516671 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.516672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.517904 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.517907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.518403 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.518405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.518557 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.519007 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.519017 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.520380 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.521875 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.521880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.521882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.522144 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.522326 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.522327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.522327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.522432 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.522681 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.522686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.522807 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.522895 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.523024 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260927 03:53:49.523653 2680 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38607
I20260927 03:53:49.523728 2680 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:49.523906 2680 heartbeater.cc:511] Master 127.1.50.126:38607 requested a full tablet report, sending...
I20260927 03:53:49.524272 2337 ts_manager.cc:194] Registered new tserver with Master: be8abf885dda4495ab9bfe779419f141 (127.1.50.66:37927)
I20260927 03:53:49.524663 2337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:40473
I20260927 03:53:49.530052 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.524685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:41053 (local address 127.1.50.126:38607)
0927 03:53:49.524765 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.524767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.524796 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:53:49.524853 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.524854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.524863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.524919 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.525302 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.525782 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.526523 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.526710 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.526764 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.527598 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.527599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.527599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.527605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.528872 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.529222 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.529223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.529224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.529283 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.529507 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.529508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.529509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.529711 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.529776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.529889 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.529951 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 03:53:49.530079 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.524623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.66:41053)
0927 03:53:49.524743 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.524753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.524809 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.524921 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.524921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.525116 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.525181 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.525183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.525881 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.525882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.526451 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.526453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.526777 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.526777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.526837 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.527232 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.527236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.527496 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.528961 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.528962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.528963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.529136 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.529365 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.529366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.529366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.529426 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.529772 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.529772 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.529809 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.529856 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.529940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260927 03:53:49.530448 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2549
I20260927 03:53:49.530525 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:53:49.530692 1225 external_mini_cluster.cc:1428] Setting key 355648c94b7cc1c3770bfab1e3636084
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:53:49.557047 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38607
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:49.637480 2689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:49.637701 2689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:49.637724 2689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:49.638917 2689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:49.638962 2689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:49.638981 2689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:49.638998 2689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:53:49.640398 2689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:49.640605 2689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:49.640786 2689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:49.645838 2697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:49.645808 2695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.645860 2689 server_base.cc:1034] running on GCE node
W20260927 03:53:49.645803 2694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:49.646258 2689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:49.646509 2689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:49.647682 2689 hybrid_clock.cc:648] HybridClock initialized: now 1790481229647664 us; error 30 us; skew 500 ppm
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:49.650112 2689 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:53:49.650512 2689 webserver.cc:460] Webserver started at http://127.1.50.67:34797/ using document root <none> and password file <none>
I20260927 03:53:49.650710 2689 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:49.650749 2689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:49.650840 2689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:49.651891 2689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "72be78514a4d436ca4291abb04837166"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "7cf3e445fc99326da70a307b5d74d4a9"
server_key_iv: "44c51f9967ce9d16644a62400899be46"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.652184 2689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "72be78514a4d436ca4291abb04837166"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "7cf3e445fc99326da70a307b5d74d4a9"
server_key_iv: "44c51f9967ce9d16644a62400899be46"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.653373 2689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:53:49.654088 2704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.654286 2689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:49.654362 2689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "72be78514a4d436ca4291abb04837166"
format_stamp: "Formatted at 2026-09-27 03:53:49 on dist-test-slave-96fg"
server_key: "7cf3e445fc99326da70a307b5d74d4a9"
server_key_iv: "44c51f9967ce9d16644a62400899be46"
server_key_version: "encryptionkey@0"
I20260927 03:53:49.654430 2689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:49.667616 2689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:49.669710 2689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:49.669847 2689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:49.670431 2689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:49.670475 2689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.670506 2689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:49.670522 2689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.677187 2689 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:36447
I20260927 03:53:49.677274 2819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:36447 every 8 connection(s)
I20260927 03:53:49.677597 2689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:53:49.679797 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.670852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:44117 (local address 127.1.50.126:38607)
0927 03:53:49.670914 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.670917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.671096 (+ 179us) server_negotiation.cc:407] Connection header received
0927 03:53:49.671571 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.671572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.671585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.671665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.672273 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.673016 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.674471 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.674765 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.674857 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.677120 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.677121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.677121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.677128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.678543 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.678920 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.678922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.678922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.678981 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.679198 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.679199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.679199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.679313 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.679367 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.679628 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.679709 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:53:49.679955 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.670825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.67:44117)
0927 03:53:49.670975 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.671015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.671486 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.671734 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.671738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.671997 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.672199 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.672204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.673116 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.673117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.674400 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.674403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.674888 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.674889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.675047 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.675571 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.675580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.677001 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.678628 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.678635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.678637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.678853 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.679035 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.679035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.679036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.679143 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.679365 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.679371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.679507 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.679595 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.679749 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260927 03:53:49.680419 2820 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38607
I20260927 03:53:49.680498 2820 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:49.680684 2820 heartbeater.cc:511] Master 127.1.50.126:38607 requested a full tablet report, sending...
I20260927 03:53:49.681087 2336 ts_manager.cc:194] Registered new tserver with Master: 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:49.681597 2336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:44117
I20260927 03:53:49.681700 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2689
I20260927 03:53:49.681777 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:53:49.681897 1225 external_mini_cluster.cc:1428] Setting key 56d9ce6fd6b318478d201a51775efe83
I20260927 03:53:49.682727 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:53:49.687150 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.681419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:41161 (local address 127.1.50.126:38607)
0927 03:53:49.681453 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.681455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.681552 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:53:49.681597 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.681598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.681607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.681664 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.681960 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.682688 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.683377 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.683633 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.683698 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.684603 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.684604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.684604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.684609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.685970 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.686332 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.686333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.686333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.686393 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.686640 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.686641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.686641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.686755 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.686812 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.686953 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.687049 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:53:49.687282 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.681372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.1.50.67:41161)
0927 03:53:49.681492 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.681501 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.681551 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.681680 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.681681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.681841 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.681896 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.681898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.682763 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.682764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.683303 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.683304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.683700 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.683701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.683756 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.684131 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.684136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.684515 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.686074 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.686075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.686076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.686268 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.686503 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.686504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.686505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.686579 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.686809 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.686810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.686855 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.686924 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.687064 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.713796 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.705980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.1.50.126:38607)
0927 03:53:49.706059 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.706061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.706073 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:53:49.706103 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.706104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.706115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.706196 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.706953 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.707779 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.708814 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.709010 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.709080 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.711643 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.711645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.711646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.711654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.712882 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.713218 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.713219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.713219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.713264 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.713404 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.713405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.713405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.713496 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.713562 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.713642 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.713702 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:53:49.715335 2336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:49.716794 2336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:49.723222 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.1.50.66:37927)
0927 03:53:49.719614 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.719620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.719628 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:49.719660 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.719662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.719677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.719749 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.719919 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.720984 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722285 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.722819 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722885 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.722949 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.723081 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.723123 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260927 03:53:49.723212 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:37927 (local address 127.0.0.1:56252)
0927 03:53:49.719418 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.719425 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.719457 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.719809 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.719810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.719812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.719851 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.719852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.721078 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.721079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.722213 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.722214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.722895 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.722978 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.722993 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.723062 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.723131 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":88,"threads_started":1}
I20260927 03:53:49.723363 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:33227 (local address 127.0.0.1:38702)
0927 03:53:49.719210 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.719258 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.719324 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.719898 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.719898 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.719900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.719946 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.719947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.720757 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.720758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.721976 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.721979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.723028 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.723029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.723114 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.723118 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.723162 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.723207 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":44,"threads_started":1}
I20260927 03:53:49.724239 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.1.50.65:33227)
0927 03:53:49.719692 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.719696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.719705 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:53:49.719745 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.719747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.719762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.719832 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.720012 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.720682 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722133 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.722876 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722945 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.722997 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.723254 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.723790 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":69,"threads_started":1}
I20260927 03:53:49.724172 2614 tablet_service.cc:1467] Processing CreateTablet for tablet 099204d2e7e34b28990405a51890980b (DEFAULT_TABLE table=test-table [id=e93252582678415bb7c8c8282d5b44f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.724566 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099204d2e7e34b28990405a51890980b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.724608 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36447 (local address 127.0.0.1:49968)
0927 03:53:49.719262 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.719270 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.719324 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.719588 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.719591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.719593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.719729 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.719731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.722097 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.722098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.723172 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.723174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.724301 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.724303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.724380 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.724385 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.724446 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.724486 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260927 03:53:49.725013 2474 tablet_service.cc:1467] Processing CreateTablet for tablet 099204d2e7e34b28990405a51890980b (DEFAULT_TABLE table=test-table [id=e93252582678415bb7c8c8282d5b44f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.725318 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099204d2e7e34b28990405a51890980b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.726220 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.719263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.1.50.67:36447)
0927 03:53:49.719407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.719414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.719432 (+ 18us) server_negotiation.cc:407] Connection header received
0927 03:53:49.719477 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.719479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.719495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.719581 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.719808 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.720984 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.723254 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.724192 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.724264 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.724329 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.724467 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.726120 (+ 1653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260927 03:53:49.726992 2754 tablet_service.cc:1467] Processing CreateTablet for tablet 099204d2e7e34b28990405a51890980b (DEFAULT_TABLE table=test-table [id=e93252582678415bb7c8c8282d5b44f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.727298 2754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099204d2e7e34b28990405a51890980b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.727883 2843 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Bootstrap starting.
I20260927 03:53:49.728423 2843 tablet_bootstrap.cc:654] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.728564 2844 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Bootstrap starting.
I20260927 03:53:49.728906 2843 log.cc:824] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:49.729159 2844 tablet_bootstrap.cc:654] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.729432 2844 log.cc:824] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:49.729779 2843 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: No bootstrap required, opened a new log
I20260927 03:53:49.729835 2843 ts_tablet_manager.cc:1397] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:49.730136 2847 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Bootstrap starting.
I20260927 03:53:49.730307 2844 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: No bootstrap required, opened a new log
I20260927 03:53:49.730425 2844 ts_tablet_manager.cc:1397] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:53:49.730942 2847 tablet_bootstrap.cc:654] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.731164 2847 log.cc:824] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:49.731712 2847 tablet_bootstrap.cc:492] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: No bootstrap required, opened a new log
I20260927 03:53:49.731762 2847 ts_tablet_manager.cc:1397] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:53:49.731945 2843 raft_consensus.cc:348] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.732110 2843 raft_consensus.cc:374] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.732148 2843 raft_consensus.cc:729] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8abf885dda4495ab9bfe779419f141, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.732256 2843 consensus_queue.cc:260] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.732508 2843 ts_tablet_manager.cc:1428] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:53:49.732553 2680 heartbeater.cc:503] Master 127.1.50.126:38607 was elected leader, sending a full tablet report...
I20260927 03:53:49.732888 2844 raft_consensus.cc:348] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.733100 2844 raft_consensus.cc:374] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.733177 2844 raft_consensus.cc:729] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af22044f24c04bb2abd6ede44da4ad07, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.733253 2847 raft_consensus.cc:348] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.733326 2844 consensus_queue.cc:260] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.733404 2847 raft_consensus.cc:374] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.733444 2847 raft_consensus.cc:729] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72be78514a4d436ca4291abb04837166, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.733578 2844 ts_tablet_manager.cc:1428] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:53:49.733549 2847 consensus_queue.cc:260] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.733685 2540 heartbeater.cc:503] Master 127.1.50.126:38607 was elected leader, sending a full tablet report...
I20260927 03:53:49.733791 2847 ts_tablet_manager.cc:1428] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:53:49.733842 2820 heartbeater.cc:503] Master 127.1.50.126:38607 was elected leader, sending a full tablet report...
I20260927 03:53:49.748611 2849 raft_consensus.cc:482] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:49.748721 2849 raft_consensus.cc:504] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.749002 2849 leader_election.cc:288] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:49.751780 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.749233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:48883 (local address 127.1.50.65:33227)
0927 03:53:49.749286 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.749288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.749298 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:53:49.749325 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.749325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.749337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.749388 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.749559 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.750153 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751117 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.751464 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751523 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.751535 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.751679 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.751713 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:53:49.751781 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.749321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:39395 (local address 127.1.50.67:36447)
0927 03:53:49.749359 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.749360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.749405 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:53:49.749456 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.749457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.749468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.749553 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.749655 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.750152 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751102 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.751467 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751534 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.751544 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.751677 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.751710 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:53:49.751791 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.749173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:33227 (local address 127.1.50.66:48883)
0927 03:53:49.749229 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.749240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.749280 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.749425 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.749426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.749428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.749497 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.749499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.750239 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.750240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.751031 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.751033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.751533 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.751616 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.751618 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.751663 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.751704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":16}
I20260927 03:53:49.751801 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.749194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36447 (local address 127.1.50.66:39395)
0927 03:53:49.749361 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.749369 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.749411 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.749530 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.749531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.749533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.749603 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.749605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.750221 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.750222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.751022 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.751024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.751536 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.751537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.751618 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.751621 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.751667 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.751714 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":29,"thread_start_us":106,"threads_started":1}
I20260927 03:53:49.752104 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099204d2e7e34b28990405a51890980b" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af22044f24c04bb2abd6ede44da4ad07" is_pre_election: true
I20260927 03:53:49.752190 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099204d2e7e34b28990405a51890980b" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166" is_pre_election: true
I20260927 03:53:49.752259 2494 raft_consensus.cc:2454] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 0.
I20260927 03:53:49.752337 2774 raft_consensus.cc:2454] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 0.
I20260927 03:53:49.752480 2567 leader_election.cc:302] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af22044f24c04bb2abd6ede44da4ad07, be8abf885dda4495ab9bfe779419f141; no voters:
I20260927 03:53:49.752599 2849 raft_consensus.cc:2781] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:49.752650 2849 raft_consensus.cc:482] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:49.752671 2849 raft_consensus.cc:3037] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.753537 2849 raft_consensus.cc:504] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.753660 2849 leader_election.cc:288] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 election: Requested vote from peers af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:49.753804 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099204d2e7e34b28990405a51890980b" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166"
I20260927 03:53:49.753837 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099204d2e7e34b28990405a51890980b" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af22044f24c04bb2abd6ede44da4ad07"
I20260927 03:53:49.753881 2774 raft_consensus.cc:3037] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.753897 2494 raft_consensus.cc:3037] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.754710 2494 raft_consensus.cc:2454] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 1.
I20260927 03:53:49.754729 2774 raft_consensus.cc:2454] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 1.
I20260927 03:53:49.754846 2567 leader_election.cc:302] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af22044f24c04bb2abd6ede44da4ad07, be8abf885dda4495ab9bfe779419f141; no voters:
I20260927 03:53:49.754922 2849 raft_consensus.cc:2781] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:49.755057 2849 raft_consensus.cc:686] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 1 LEADER]: Becoming Leader. State: Replica: be8abf885dda4495ab9bfe779419f141, State: Running, Role: LEADER
I20260927 03:53:49.755133 2849 consensus_queue.cc:237] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.755859 2337 catalog_manager.cc:5495] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 reported cstate change: term changed from 0 to 1, leader changed from <none> to be8abf885dda4495ab9bfe779419f141 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:49.769450 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.767059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.1.50.66:37927)
0927 03:53:49.767113 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.767115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.767123 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:49.767202 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.767203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.767224 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.767278 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:49.767399 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.767878 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.768675 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.768882 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.768938 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.769125 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:49.769224 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:49.769290 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.769332 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.769356 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 03:53:49.771157 2681 tablet.cc:2369] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:53:49.771176 2774 raft_consensus.cc:1261] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Refusing update from remote peer be8abf885dda4495ab9bfe779419f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:49.771211 2494 raft_consensus.cc:1261] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Refusing update from remote peer be8abf885dda4495ab9bfe779419f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:49.771515 2849 consensus_queue.cc:1035] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:49.771652 2853 consensus_queue.cc:1035] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:49.774718 2857 mvcc.cc:205] Tried to move back new op lower bound from 7333811117140254720 to 7333811117077598208. Current Snapshot: MvccSnapshot[applied={T|T < 7333811117140254720}]
I20260927 03:53:49.777449 2335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:49.777570 2335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:49.778409 2335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Sending DeleteTablet for 3 replicas of tablet 099204d2e7e34b28990405a51890980b
I20260927 03:53:49.778843 2754 tablet_service.cc:1514] Processing DeleteTablet for tablet 099204d2e7e34b28990405a51890980b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:49 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:49968
I20260927 03:53:49.778816 2474 tablet_service.cc:1514] Processing DeleteTablet for tablet 099204d2e7e34b28990405a51890980b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:49 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38702
I20260927 03:53:49.778841 2614 tablet_service.cc:1514] Processing DeleteTablet for tablet 099204d2e7e34b28990405a51890980b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:49 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:56252
I20260927 03:53:49.779084 2335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:49.779237 2335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:49.779709 2867 tablet_replica.cc:330] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: stopping tablet replica
I20260927 03:53:49.779822 2867 raft_consensus.cc:2229] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:49.779924 2867 raft_consensus.cc:2258] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:49.780125 2866 tablet_replica.cc:330] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: stopping tablet replica
I20260927 03:53:49.780220 2866 raft_consensus.cc:2229] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:53:49.780340 2866 raft_consensus.cc:2258] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:49.780429 2614 tablet_service.cc:1467] Processing CreateTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 (DEFAULT_TABLE table=test-table [id=d0926e3b6da94237b3479591b77f0b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.780464 2754 tablet_service.cc:1467] Processing CreateTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 (DEFAULT_TABLE table=test-table [id=d0926e3b6da94237b3479591b77f0b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.780545 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c1ab9789b449b89747f2eb93543ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.780550 2754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c1ab9789b449b89747f2eb93543ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.780821 2866 ts_tablet_manager.cc:1905] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:49.781910 2847 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Bootstrap starting.
I20260927 03:53:49.781924 2843 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Bootstrap starting.
I20260927 03:53:49.782122 2867 ts_tablet_manager.cc:1905] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:49.782318 2843 tablet_bootstrap.cc:654] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.782439 2847 tablet_bootstrap.cc:654] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.783007 2843 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: No bootstrap required, opened a new log
I20260927 03:53:49.782996 2474 tablet_service.cc:1467] Processing CreateTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 (DEFAULT_TABLE table=test-table [id=d0926e3b6da94237b3479591b77f0b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:49.783048 2843 ts_tablet_manager.cc:1397] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.783092 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c1ab9789b449b89747f2eb93543ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.783172 2843 raft_consensus.cc:348] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.783243 2843 raft_consensus.cc:374] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.783262 2843 raft_consensus.cc:729] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8abf885dda4495ab9bfe779419f141, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.783298 2843 consensus_queue.cc:260] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.783368 2843 ts_tablet_manager.cc:1428] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.784319 2844 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Bootstrap starting.
I20260927 03:53:49.784375 2866 ts_tablet_manager.cc:1918] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:53:49.784438 2866 log.cc:1196] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/099204d2e7e34b28990405a51890980b
I20260927 03:53:49.784651 2844 tablet_bootstrap.cc:654] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.784680 2866 ts_tablet_manager.cc:1939] T 099204d2e7e34b28990405a51890980b P be8abf885dda4495ab9bfe779419f141: Deleting consensus metadata
I20260927 03:53:49.785072 2321 catalog_manager.cc:4841] TS be8abf885dda4495ab9bfe779419f141 (127.1.50.66:37927): tablet 099204d2e7e34b28990405a51890980b (table test-table [id=e93252582678415bb7c8c8282d5b44f8]) successfully deleted
I20260927 03:53:49.785147 2844 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: No bootstrap required, opened a new log
I20260927 03:53:49.785197 2844 ts_tablet_manager.cc:1397] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.785338 2844 raft_consensus.cc:348] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.785424 2844 raft_consensus.cc:374] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.785444 2844 raft_consensus.cc:729] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af22044f24c04bb2abd6ede44da4ad07, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.785485 2844 consensus_queue.cc:260] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.785560 2844 ts_tablet_manager.cc:1428] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:49.785882 2867 ts_tablet_manager.cc:1918] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:53:49.785957 2867 log.cc:1196] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/099204d2e7e34b28990405a51890980b
I20260927 03:53:49.786253 2867 ts_tablet_manager.cc:1939] T 099204d2e7e34b28990405a51890980b P af22044f24c04bb2abd6ede44da4ad07: Deleting consensus metadata
I20260927 03:53:49.786443 2868 tablet_replica.cc:330] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: stopping tablet replica
I20260927 03:53:49.786580 2868 raft_consensus.cc:2229] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:49.786752 2323 catalog_manager.cc:4841] TS af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227): tablet 099204d2e7e34b28990405a51890980b (table test-table [id=e93252582678415bb7c8c8282d5b44f8]) successfully deleted
I20260927 03:53:49.786783 2868 raft_consensus.cc:2258] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:49.787274 2868 ts_tablet_manager.cc:1905] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:49.788792 2868 ts_tablet_manager.cc:1918] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:53:49.788868 2868 log.cc:1196] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/099204d2e7e34b28990405a51890980b
I20260927 03:53:49.789100 2868 ts_tablet_manager.cc:1939] T 099204d2e7e34b28990405a51890980b P 72be78514a4d436ca4291abb04837166: Deleting consensus metadata
I20260927 03:53:49.789959 2324 catalog_manager.cc:4841] TS 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447): tablet 099204d2e7e34b28990405a51890980b (table test-table [id=e93252582678415bb7c8c8282d5b44f8]) successfully deleted
I20260927 03:53:49.794452 2847 tablet_bootstrap.cc:492] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: No bootstrap required, opened a new log
I20260927 03:53:49.794510 2847 ts_tablet_manager.cc:1397] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260927 03:53:49.794665 2847 raft_consensus.cc:348] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.794740 2847 raft_consensus.cc:374] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.794759 2847 raft_consensus.cc:729] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72be78514a4d436ca4291abb04837166, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.794798 2847 consensus_queue.cc:260] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.794883 2847 ts_tablet_manager.cc:1428] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:49.842130 2850 raft_consensus.cc:482] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:49.842238 2850 raft_consensus.cc:504] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.842543 2850 leader_election.cc:288] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be8abf885dda4495ab9bfe779419f141 (127.1.50.66:37927), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:49.845811 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.842714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36447 (local address 127.1.50.65:42209)
0927 03:53:49.842772 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.842783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.842822 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.842944 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.842945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.842946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.843012 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.843015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.843755 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.843756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.844837 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.844840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.845569 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.845570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.845639 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.845642 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.845692 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.845734 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:53:49.845817 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.842767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:42209 (local address 127.1.50.67:36447)
0927 03:53:49.842819 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.842821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.842829 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:49.842872 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.842873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.842882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.842921 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.843097 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.843675 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.844939 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.845480 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.845574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.845587 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.845713 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.845756 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:53:49.845947 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.842822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:43489 (local address 127.1.50.66:37927)
0927 03:53:49.842866 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.842868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.843065 (+ 197us) server_negotiation.cc:407] Connection header received
0927 03:53:49.843100 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.843101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.843110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.843179 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:49.843316 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.843975 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.845182 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.845532 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.845603 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.845613 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.845836 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.845877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:53:49.846197 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.842719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:37927 (local address 127.1.50.65:43489)
0927 03:53:49.842994 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.843003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.843055 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.843185 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.843185 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.843187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:49.843252 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.843254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.844067 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.844068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.845106 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.845107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.845661 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.845662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.845739 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.845743 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.846067 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.846112 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":159,"thread_start_us":58,"threads_started":1}
I20260927 03:53:49.846329 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c1ab9789b449b89747f2eb93543ad4" candidate_uuid: "af22044f24c04bb2abd6ede44da4ad07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166" is_pre_election: true
I20260927 03:53:49.846416 2774 raft_consensus.cc:2454] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af22044f24c04bb2abd6ede44da4ad07 in term 0.
I20260927 03:53:49.846441 2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c1ab9789b449b89747f2eb93543ad4" candidate_uuid: "af22044f24c04bb2abd6ede44da4ad07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8abf885dda4495ab9bfe779419f141" is_pre_election: true
I20260927 03:53:49.846565 2634 raft_consensus.cc:2454] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af22044f24c04bb2abd6ede44da4ad07 in term 0.
I20260927 03:53:49.846626 2428 leader_election.cc:302] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72be78514a4d436ca4291abb04837166, af22044f24c04bb2abd6ede44da4ad07; no voters:
I20260927 03:53:49.846752 2850 raft_consensus.cc:2781] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:49.846792 2850 raft_consensus.cc:482] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:49.846812 2850 raft_consensus.cc:3037] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.847496 2850 raft_consensus.cc:504] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.847625 2850 leader_election.cc:288] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [CANDIDATE]: Term 1 election: Requested vote from peers be8abf885dda4495ab9bfe779419f141 (127.1.50.66:37927), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:49.847756 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c1ab9789b449b89747f2eb93543ad4" candidate_uuid: "af22044f24c04bb2abd6ede44da4ad07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166"
I20260927 03:53:49.847829 2774 raft_consensus.cc:3037] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.848382 2774 raft_consensus.cc:2454] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af22044f24c04bb2abd6ede44da4ad07 in term 1.
I20260927 03:53:49.848479 2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c1ab9789b449b89747f2eb93543ad4" candidate_uuid: "af22044f24c04bb2abd6ede44da4ad07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8abf885dda4495ab9bfe779419f141"
I20260927 03:53:49.848536 2428 leader_election.cc:302] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72be78514a4d436ca4291abb04837166, af22044f24c04bb2abd6ede44da4ad07; no voters:
I20260927 03:53:49.848551 2634 raft_consensus.cc:3037] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:49.848609 2850 raft_consensus.cc:2781] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:49.848704 2850 raft_consensus.cc:686] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 1 LEADER]: Becoming Leader. State: Replica: af22044f24c04bb2abd6ede44da4ad07, State: Running, Role: LEADER
I20260927 03:53:49.848769 2850 consensus_queue.cc:237] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.849308 2634 raft_consensus.cc:2454] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af22044f24c04bb2abd6ede44da4ad07 in term 1.
I20260927 03:53:49.849589 2336 catalog_manager.cc:5495] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 reported cstate change: term changed from 0 to 1, leader changed from <none> to af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "af22044f24c04bb2abd6ede44da4ad07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
W20260927 03:53:49.872483 2541 tablet.cc:2369] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:53:49 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:49.886488 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.878844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.1.50.126:38607)
0927 03:53:49.878895 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:49.878897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:49.879006 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:53:49.879073 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:49.879073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:49.879082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:49.879131 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:49.879716 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.880234 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.881262 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.881477 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.881536 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.883716 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:49.883718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:49.883718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:49.883725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:49.885212 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.885583 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.885584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.885584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.885642 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:49.885884 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:49.885885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:49.885885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:49.886051 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:49.886114 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:49.886264 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:53:49.886366 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":27}
I20260927 03:53:49.886590 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:49.878753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38607 (local address 127.0.0.1:47522)
0927 03:53:49.878915 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:49.878927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:49.879010 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:49.879154 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:49.879155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:49.879536 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:49.879603 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.879605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.880308 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.880309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.881167 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:49.881169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:49.881541 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:49.881542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:49.881601 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:49.882194 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:49.882198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:49.883639 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:49.885301 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.885308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.885310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.885510 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.885726 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:49.885727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:49.885727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:49.885791 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:49.886109 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:49.886116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:49.886160 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:53:49.886254 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:53:49.886435 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":69,"thread_start_us":73,"threads_started":1}
I20260927 03:53:49.888633 2336 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:47522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:53:49.888850 2336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:49.890308 2614 tablet_service.cc:1467] Processing CreateTablet for tablet bfeff02b9766455eaacc6dee4305dcf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=447a7fae6b8744cab0f2067d6ada3674]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:49.890381 2754 tablet_service.cc:1467] Processing CreateTablet for tablet bfeff02b9766455eaacc6dee4305dcf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=447a7fae6b8744cab0f2067d6ada3674]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:49.890475 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeff02b9766455eaacc6dee4305dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.890489 2754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeff02b9766455eaacc6dee4305dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.891892 2847 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: Bootstrap starting.
I20260927 03:53:49.891983 2474 tablet_service.cc:1467] Processing CreateTablet for tablet bfeff02b9766455eaacc6dee4305dcf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=447a7fae6b8744cab0f2067d6ada3674]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:49.892133 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeff02b9766455eaacc6dee4305dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:49.892320 2847 tablet_bootstrap.cc:654] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.892627 2843 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: Bootstrap starting.
I20260927 03:53:49.892901 2843 tablet_bootstrap.cc:654] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.893268 2847 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: No bootstrap required, opened a new log
I20260927 03:53:49.893302 2844 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: Bootstrap starting.
I20260927 03:53:49.893316 2847 ts_tablet_manager.cc:1397] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.893446 2847 raft_consensus.cc:348] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.893507 2847 raft_consensus.cc:374] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.893524 2847 raft_consensus.cc:729] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72be78514a4d436ca4291abb04837166, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.893693 2844 tablet_bootstrap.cc:654] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:49.893675 2847 consensus_queue.cc:260] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.893882 2847 ts_tablet_manager.cc:1428] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:49.894009 2851 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:49.894101 2851 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:49.894457 2843 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: No bootstrap required, opened a new log
I20260927 03:53:49.894511 2843 ts_tablet_manager.cc:1397] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260927 03:53:49.894604 2843 raft_consensus.cc:348] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.894663 2843 raft_consensus.cc:374] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.894680 2843 raft_consensus.cc:729] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8abf885dda4495ab9bfe779419f141, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.894721 2843 consensus_queue.cc:260] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.894796 2843 ts_tablet_manager.cc:1428] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.894829 2853 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:49.894927 2853 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:49.895433 2844 tablet_bootstrap.cc:492] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: No bootstrap required, opened a new log
I20260927 03:53:49.895483 2844 ts_tablet_manager.cc:1397] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:49.895601 2844 raft_consensus.cc:348] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.895653 2844 raft_consensus.cc:374] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:49.895665 2844 raft_consensus.cc:729] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af22044f24c04bb2abd6ede44da4ad07, State: Initialized, Role: FOLLOWER
I20260927 03:53:49.895699 2844 consensus_queue.cc:260] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:49.895783 2844 ts_tablet_manager.cc:1428] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:49.895839 2873 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:49.895915 2873 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:53:49.928180 2821 tablet.cc:2369] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:53:50.131867 2853 raft_consensus.cc:482] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:50.131994 2853 raft_consensus.cc:504] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:50.132166 2853 leader_election.cc:288] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:50.132414 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeff02b9766455eaacc6dee4305dcf2" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af22044f24c04bb2abd6ede44da4ad07" is_pre_election: true
I20260927 03:53:50.132467 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeff02b9766455eaacc6dee4305dcf2" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166" is_pre_election: true
I20260927 03:53:50.132551 2774 raft_consensus.cc:2454] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 0.
I20260927 03:53:50.132552 2494 raft_consensus.cc:2454] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 0.
I20260927 03:53:50.132700 2568 leader_election.cc:302] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72be78514a4d436ca4291abb04837166, be8abf885dda4495ab9bfe779419f141; no voters:
I20260927 03:53:50.132808 2853 raft_consensus.cc:2781] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:50.132855 2853 raft_consensus.cc:482] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:50.132881 2853 raft_consensus.cc:3037] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:50.133659 2853 raft_consensus.cc:504] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:50.133785 2853 leader_election.cc:288] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 election: Requested vote from peers af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227), 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447)
I20260927 03:53:50.133962 2774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeff02b9766455eaacc6dee4305dcf2" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72be78514a4d436ca4291abb04837166"
I20260927 03:53:50.134081 2774 raft_consensus.cc:3037] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:50.134100 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeff02b9766455eaacc6dee4305dcf2" candidate_uuid: "be8abf885dda4495ab9bfe779419f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af22044f24c04bb2abd6ede44da4ad07"
I20260927 03:53:50.134177 2494 raft_consensus.cc:3037] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:50.134678 2774 raft_consensus.cc:2454] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 1.
I20260927 03:53:50.134737 2494 raft_consensus.cc:2454] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abf885dda4495ab9bfe779419f141 in term 1.
I20260927 03:53:50.134822 2568 leader_election.cc:302] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72be78514a4d436ca4291abb04837166, be8abf885dda4495ab9bfe779419f141; no voters:
I20260927 03:53:50.134917 2853 raft_consensus.cc:2781] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:50.134976 2853 raft_consensus.cc:686] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [term 1 LEADER]: Becoming Leader. State: Replica: be8abf885dda4495ab9bfe779419f141, State: Running, Role: LEADER
I20260927 03:53:50.135031 2853 consensus_queue.cc:237] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } }
I20260927 03:53:50.135294 2861 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: TxnStatusTablet state changed. Reason: New leader be8abf885dda4495ab9bfe779419f141. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.135378 2861 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:50.135517 2885 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:53:50.135627 2885 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:53:50.135597 2336 catalog_manager.cc:5495] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 reported cstate change: term changed from 0 to 1, leader changed from <none> to be8abf885dda4495ab9bfe779419f141 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:50.179034 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.176362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.1.50.66:37927)
0927 03:53:50.176429 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.176431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.176442 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:53:50.176505 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.176506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.176527 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.176581 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:50.176680 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.177408 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.178335 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.178563 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.178626 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.178806 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.178840 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.178882 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.178941 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.178961 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:53:50.179049 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.176269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:37927 (local address 127.0.0.1:56266)
0927 03:53:50.176365 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:50.176378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:50.176431 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:50.176587 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:50.176588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:50.176589 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:50.176634 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.176636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.177499 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.177501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.178241 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.178244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.178659 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.178660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.178714 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.178728 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.178891 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.178898 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:53:50.178939 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:53:50.178988 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 03:53:50.180099 2494 raft_consensus.cc:1261] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Refusing update from remote peer be8abf885dda4495ab9bfe779419f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:50.180336 2853 consensus_queue.cc:1035] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:50.180469 2774 raft_consensus.cc:1261] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Refusing update from remote peer be8abf885dda4495ab9bfe779419f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:50.180706 2853 consensus_queue.cc:1035] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:50.180871 2873 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: TxnStatusTablet state changed. Reason: New leader be8abf885dda4495ab9bfe779419f141. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.180965 2873 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:50.181207 2851 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: TxnStatusTablet state changed. Reason: New leader be8abf885dda4495ab9bfe779419f141. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.181317 2851 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:50.181354 2853 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.181422 2853 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:50.181492 2856 mvcc.cc:205] Tried to move back new op lower bound from 7333811118815817728 to 7333811118633447424. Current Snapshot: MvccSnapshot[applied={T|T < 7333811118815817728}]
I20260927 03:53:50.182215 2873 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.182297 2873 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P af22044f24c04bb2abd6ede44da4ad07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:50.182984 2849 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.183068 2849 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P be8abf885dda4495ab9bfe779419f141: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:50.183509 2858 mvcc.cc:205] Tried to move back new op lower bound from 7333811118815817728 to 7333811118633447424. Current Snapshot: MvccSnapshot[applied={T|T < 7333811118815817728}]
I20260927 03:53:50.183619 2851 tablet_replica.cc:439] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be8abf885dda4495ab9bfe779419f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af22044f24c04bb2abd6ede44da4ad07" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 33227 } } peers { permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 } } peers { permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 } } }
I20260927 03:53:50.183686 2851 tablet_replica.cc:442] T bfeff02b9766455eaacc6dee4305dcf2 P 72be78514a4d436ca4291abb04837166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:50.186074 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.183189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.1.50.65:33227)
0927 03:53:50.183231 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.183232 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.183238 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:53:50.183271 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.183272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.183292 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.183354 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:50.183519 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.184174 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.185216 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.185422 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.185482 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.185669 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.185805 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.185877 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.185927 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.185952 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:53:50.191277 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.189011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:46345 (local address 127.1.50.66:37927)
0927 03:53:50.189062 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.189064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.189189 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:53:50.189250 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.189251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.189262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.189312 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:50.189425 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.189877 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.190658 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.190849 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.190902 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.191050 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.191082 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.191122 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.191175 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.191195 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:53:50.191417 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.188958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:37927 (local address 127.1.50.65:46345)
0927 03:53:50.189126 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:50.189136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:50.189190 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:50.189315 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:50.189315 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:50.189317 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:50.189374 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.189376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.189940 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.189941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.190586 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.190589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.190905 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.190906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.190969 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.190985 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.191126 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.191128 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:53:50.191164 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 03:53:50.191207 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260927 03:53:50.193543 2774 raft_consensus.cc:1261] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Refusing update from remote peer af22044f24c04bb2abd6ede44da4ad07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:50.193620 2634 raft_consensus.cc:1261] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Refusing update from remote peer af22044f24c04bb2abd6ede44da4ad07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:50.193790 2873 consensus_queue.cc:1035] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72be78514a4d436ca4291abb04837166" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:50.193899 2850 consensus_queue.cc:1035] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be8abf885dda4495ab9bfe779419f141" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:50.201479 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.199179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:33227 (local address 127.1.50.66:37671)
0927 03:53:50.199324 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:50.199335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:50.199382 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:50.199490 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:50.199491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:50.199493 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:50.199537 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.199539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.200106 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.200107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.200744 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:50.200747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.201057 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.201058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:50.201109 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.201125 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.201344 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.201346 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:53:50.201379 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:53:50.201421 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260927 03:53:50.201484 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.199363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:37671 (local address 127.1.50.65:33227)
0927 03:53:50.199403 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.199404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.199422 (+ 18us) server_negotiation.cc:407] Connection header received
0927 03:53:50.199441 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.199442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.199452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.199490 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:50.199580 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.200042 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.200805 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.201003 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.201052 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.201226 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:50.201277 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:50.201340 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.201385 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.201408 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:53:50.211058 2336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:50.211162 2336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:50.211745 2336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0881f71c5ca84684a8e5a994c3900c32: Sending DeleteTablet for 3 replicas of tablet 84c1ab9789b449b89747f2eb93543ad4
I20260927 03:53:50.212033 2754 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:50 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:49968
I20260927 03:53:50.212044 2474 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:50 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38702
I20260927 03:53:50.212131 2867 tablet_replica.cc:330] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: stopping tablet replica
I20260927 03:53:50.212169 2867 raft_consensus.cc:2229] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:53:50.212145 2868 tablet_replica.cc:330] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: stopping tablet replica
I20260927 03:53:50.212221 2868 raft_consensus.cc:2229] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:50.212251 2867 raft_consensus.cc:2258] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:50.212260 2868 raft_consensus.cc:2258] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:50.212616 2868 ts_tablet_manager.cc:1905] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:50.212891 2867 ts_tablet_manager.cc:1905] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:50.214102 2614 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c1ab9789b449b89747f2eb93543ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:50 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:56252
I20260927 03:53:50.214219 2866 tablet_replica.cc:330] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: stopping tablet replica
I20260927 03:53:50.214298 2866 raft_consensus.cc:2229] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:50.214298 2867 ts_tablet_manager.cc:1918] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:50.214341 2866 raft_consensus.cc:2258] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:50.214340 2867 log.cc:1196] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/84c1ab9789b449b89747f2eb93543ad4
I20260927 03:53:50.214334 2868 ts_tablet_manager.cc:1918] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:50.214383 2868 log.cc:1196] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/84c1ab9789b449b89747f2eb93543ad4
I20260927 03:53:50.214565 2867 ts_tablet_manager.cc:1939] T 84c1ab9789b449b89747f2eb93543ad4 P af22044f24c04bb2abd6ede44da4ad07: Deleting consensus metadata
I20260927 03:53:50.214583 2868 ts_tablet_manager.cc:1939] T 84c1ab9789b449b89747f2eb93543ad4 P 72be78514a4d436ca4291abb04837166: Deleting consensus metadata
I20260927 03:53:50.214656 2866 ts_tablet_manager.cc:1905] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:50.214848 2323 catalog_manager.cc:4841] TS af22044f24c04bb2abd6ede44da4ad07 (127.1.50.65:33227): tablet 84c1ab9789b449b89747f2eb93543ad4 (table test-table [id=d0926e3b6da94237b3479591b77f0b60]) successfully deleted
I20260927 03:53:50.214951 2324 catalog_manager.cc:4841] TS 72be78514a4d436ca4291abb04837166 (127.1.50.67:36447): tablet 84c1ab9789b449b89747f2eb93543ad4 (table test-table [id=d0926e3b6da94237b3479591b77f0b60]) successfully deleted
I20260927 03:53:50.216084 2866 ts_tablet_manager.cc:1918] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:50.216146 2866 log.cc:1196] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/84c1ab9789b449b89747f2eb93543ad4
I20260927 03:53:50.216333 2866 ts_tablet_manager.cc:1939] T 84c1ab9789b449b89747f2eb93543ad4 P be8abf885dda4495ab9bfe779419f141: Deleting consensus metadata
I20260927 03:53:50.216745 2321 catalog_manager.cc:4841] TS be8abf885dda4495ab9bfe779419f141 (127.1.50.66:37927): tablet 84c1ab9789b449b89747f2eb93543ad4 (table test-table [id=d0926e3b6da94237b3479591b77f0b60]) successfully deleted
Sep 27 03:53:50 dist-test-slave-96fg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:53:50.222363 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.212866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.1.50.65:33227)
0927 03:53:50.212902 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.212904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.214230 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 03:53:50.214608 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.214609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.214621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.214651 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:50.215530 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.216001 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.216913 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.217113 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.217169 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.219777 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:50.219785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:50.219785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:50.219792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:50.221304 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:50.221609 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:50.221613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:50.221613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:50.221663 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:50.221825 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:50.221826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:50.221826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:50.221960 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:50.222140 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.222176 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.222250 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:53:50.222667 2534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38724
I20260927 03:53:50.229329 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:50.223719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.1.50.126:38607)
0927 03:53:50.223773 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:50.223775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:50.223853 (+ 78us) server_negotiation.cc:407] Connection header received
0927 03:53:50.223919 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:50.223920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:50.223929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:50.223973 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:50.224440 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.225094 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.225763 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:50.225970 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:50.226044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:50.227104 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:50.227112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:50.227112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:50.227120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:50.228358 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:50.228661 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:50.228662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:50.228662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:50.228716 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:50.228872 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:50.228873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:50.228873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:50.228983 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:50.229065 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:50.229161 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:53:50.229237 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 03:53:50.229624 2336 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47524
I20260927 03:53:50.230159 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2409
I20260927 03:53:50.236739 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2549
I20260927 03:53:50.241896 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2689
I20260927 03:53:50.245908 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2305
2026-09-27T03:53:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2261 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:50 dist-test-slave-96fg krb5kdc[2906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:50 dist-test-slave-96fg krb5kdc[2906](info): set up 2 sockets
Sep 27 03:53:50 dist-test-slave-96fg krb5kdc[2906](info): commencing operation
krb5kdc: starting...
W20260927 03:53:51.213579 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:51.330054 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43227
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:43227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:51.409085 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:51.409307 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:51.409336 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:51.410576 2919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:51.410619 2919 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:53:51.410637 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:51.410653 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:51.410669 2919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:51.410686 2919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:51.412245 2919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:43227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:51.412487 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:51.412699 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:51.417474 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.417457 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.417721 2919 server_base.cc:1034] running on GCE node
W20260927 03:53:51.417457 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.418045 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:51.418351 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:51.419528 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790481231419514 us; error 34 us; skew 500 ppm
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:51.422194 2919 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:51.422672 2919 webserver.cc:460] Webserver started at http://127.1.50.126:37079/ using document root <none> and password file <none>
I20260927 03:53:51.422899 2919 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:51.422948 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:51.423085 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:51.424140 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "dd23d002619d44b2a044386193399d60"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "bd5c4530b1d9203883326505483d0baa"
server_key_iv: "286630bd70a5b6a7713657a91785e932"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.424476 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "dd23d002619d44b2a044386193399d60"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "bd5c4530b1d9203883326505483d0baa"
server_key_iv: "286630bd70a5b6a7713657a91785e932"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.425824 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:53:51.426729 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.426932 2919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:51.427019 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "dd23d002619d44b2a044386193399d60"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "bd5c4530b1d9203883326505483d0baa"
server_key_iv: "286630bd70a5b6a7713657a91785e932"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.427119 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:51.456784 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:51.458881 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:51.459026 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:51.463335 2919 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:43227
I20260927 03:53:51.463395 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:43227 every 8 connection(s)
I20260927 03:53:51.463691 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:51.464301 2997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:51.464926 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 2919
I20260927 03:53:51.465015 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:51.465174 1225 external_mini_cluster.cc:1428] Setting key 97766f1a9bf30a12a9184f2f62172180
I20260927 03:53:51.466933 2997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Bootstrap starting.
I20260927 03:53:51.467484 2997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:51.467730 2997 log.cc:824] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:51.468420 2997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: No bootstrap required, opened a new log
I20260927 03:53:51.470296 2997 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } }
I20260927 03:53:51.470429 2997 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:51.470463 2997 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd23d002619d44b2a044386193399d60, State: Initialized, Role: FOLLOWER
I20260927 03:53:51.470589 2997 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } }
I20260927 03:53:51.470660 2997 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:51.470690 2997 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:51.470736 2997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:51.471293 2997 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } }
I20260927 03:53:51.471405 2997 leader_election.cc:302] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd23d002619d44b2a044386193399d60; no voters:
I20260927 03:53:51.471572 2997 leader_election.cc:288] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:51.471686 3002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:51.471832 2997 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:51.471837 3002 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [term 1 LEADER]: Becoming Leader. State: Replica: dd23d002619d44b2a044386193399d60, State: Running, Role: LEADER
I20260927 03:53:51.471946 3002 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } }
I20260927 03:53:51.472285 3003 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd23d002619d44b2a044386193399d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } } }
I20260927 03:53:51.472363 3003 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:51.472507 3004 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd23d002619d44b2a044386193399d60. Latest consensus state: current_term: 1 leader_uuid: "dd23d002619d44b2a044386193399d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23d002619d44b2a044386193399d60" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43227 } } }
I20260927 03:53:51.472564 3004 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:51.472678 3010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:51.473148 3010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:51.474562 3010 catalog_manager.cc:1347] Generated new cluster ID: 903c036143fd4da5ae72eaa0818c3b34
I20260927 03:53:51.474611 3010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:53:51.475852 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.465654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.1.50.126:43227)
0927 03:53:51.465865 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.465873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.465899 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:53:51.466346 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.466354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.466407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.466531 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.467144 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.468228 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.469012 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.469437 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.469531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.472296 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.472300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.472301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.472314 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.474278 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.474697 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.474708 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.474709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.474807 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.475015 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.475016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.475016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.475262 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.475409 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.475517 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.475654 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":123,"threads_started":1}
I20260927 03:53:51.483861 3010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:51.484448 3010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:51.490301 3010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Generated new TSK 0
I20260927 03:53:51.490504 3010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:51.522063 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43227
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:51.606001 3023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:51.606284 3023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:51.606320 3023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:51.607568 3023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:51.607613 3023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:51.607641 3023 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:53:51.607656 3023 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:51.607679 3023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:53:51.609177 3023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:51.609421 3023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:51.609647 3023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:51.614962 3029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.614956 3028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.614959 3031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.615303 3023 server_base.cc:1034] running on GCE node
I20260927 03:53:51.615495 3023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:51.615708 3023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:51.616868 3023 hybrid_clock.cc:648] HybridClock initialized: now 1790481231616854 us; error 28 us; skew 500 ppm
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:51.619534 3023 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:53:51.620009 3023 webserver.cc:460] Webserver started at http://127.1.50.65:43331/ using document root <none> and password file <none>
I20260927 03:53:51.620236 3023 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:51.620285 3023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:51.620409 3023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:51.621521 3023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "95d7882f806f27648db85b123443bef0"
server_key_iv: "f4e5ee594ce11eb1c60b7cbd08737f21"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.621871 3023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "95d7882f806f27648db85b123443bef0"
server_key_iv: "f4e5ee594ce11eb1c60b7cbd08737f21"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.623216 3023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:51.624081 3038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.624244 3023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:51.624329 3023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "95d7882f806f27648db85b123443bef0"
server_key_iv: "f4e5ee594ce11eb1c60b7cbd08737f21"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.624397 3023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:51.652390 3023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:51.654799 3023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:51.654958 3023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:51.655474 3023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:51.655524 3023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.655553 3023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:51.655582 3023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:51.662307 3023 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:36289
I20260927 03:53:51.662364 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:36289 every 8 connection(s)
I20260927 03:53:51.662731 3023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:53:51.665030 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.655979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55859 (local address 127.1.50.126:43227)
0927 03:53:51.656022 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.656024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.656245 (+ 221us) server_negotiation.cc:407] Connection header received
0927 03:53:51.656818 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.656820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.656832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.656882 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.657757 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.658436 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.659445 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.659667 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.659725 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.661943 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.661945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.661945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.661954 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.663526 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.663970 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.663971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.663971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.664020 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.664328 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.664329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.664330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.664460 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.664532 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.664882 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.664963 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:53:51.665211 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.655976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.65:55859)
0927 03:53:51.656125 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.656161 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.656704 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.656972 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.656977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.657362 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.657660 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.657666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.658528 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.658529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.659357 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.659359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.659751 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.659752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.659879 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.660402 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.660410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.661829 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.663630 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.663635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.663637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.663903 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.664066 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.664067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.664067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.664163 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.664557 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.664563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.664747 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.664856 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.664991 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260927 03:53:51.665702 3154 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43227
I20260927 03:53:51.665786 3154 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:51.665972 3154 heartbeater.cc:511] Master 127.1.50.126:43227 requested a full tablet report, sending...
I20260927 03:53:51.666621 2950 ts_manager.cc:194] Registered new tserver with Master: 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289)
I20260927 03:53:51.667296 2950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:55859
I20260927 03:53:51.667428 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3023
I20260927 03:53:51.667522 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:53:51.667718 1225 external_mini_cluster.cc:1428] Setting key bffda205aa450d4ea79271381e6994da
I20260927 03:53:51.672950 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.666865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.65:59081)
0927 03:53:51.666985 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.666995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.667049 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.667150 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.667150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.667375 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.667440 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.667442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.668548 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.668549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.669269 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.669271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.669687 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.669689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.669777 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.670257 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.670261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.670559 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.671986 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.671987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.671987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.672164 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.672335 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.672336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.672336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.672397 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.672636 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.672637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.672683 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.672740 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.672850 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260927 03:53:51.672945 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.666935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:59081 (local address 127.1.50.126:43227)
0927 03:53:51.666979 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.666981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.667037 (+ 56us) server_negotiation.cc:407] Connection header received
0927 03:53:51.667093 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.667094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.667104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.667147 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.667705 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.668453 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.669355 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.669553 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.669676 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.670641 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.670642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.670642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.670649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.671914 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.672225 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.672227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.672227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.672282 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.672446 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.672446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.672447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.672581 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.672634 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.672771 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.672849 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:53:51.693403 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43227
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:51.772961 3163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:51.773160 3163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:51.773185 3163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:51.774333 3163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:51.774369 3163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:51.774389 3163 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:53:51.774398 3163 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:51.774415 3163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:53:51.775775 3163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:51.775961 3163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:51.776139 3163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:51.781078 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.781083 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.781088 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.781243 3163 server_base.cc:1034] running on GCE node
I20260927 03:53:51.781514 3163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:51.781807 3163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:51.782958 3163 hybrid_clock.cc:648] HybridClock initialized: now 1790481231782929 us; error 46 us; skew 500 ppm
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:51.785651 3163 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:53:51.786057 3163 webserver.cc:460] Webserver started at http://127.1.50.66:33439/ using document root <none> and password file <none>
I20260927 03:53:51.786233 3163 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:51.786270 3163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:51.786360 3163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:51.787328 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "c174cd58025343b584d61efe59e220a4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "cd7da3aa0f1a0dcdd38a640b18c594c9"
server_key_iv: "ec994958b56a14cc517943359b48887f"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.787631 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "c174cd58025343b584d61efe59e220a4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "cd7da3aa0f1a0dcdd38a640b18c594c9"
server_key_iv: "ec994958b56a14cc517943359b48887f"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.788879 3163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:53:51.789562 3178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.789696 3163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:51.789781 3163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "c174cd58025343b584d61efe59e220a4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "cd7da3aa0f1a0dcdd38a640b18c594c9"
server_key_iv: "ec994958b56a14cc517943359b48887f"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.789849 3163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:51.806221 3163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:51.808465 3163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:51.808607 3163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:51.809144 3163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:51.809190 3163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.809214 3163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:51.809230 3163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:51.816236 3163 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:45483
I20260927 03:53:51.816303 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:45483 every 8 connection(s)
I20260927 03:53:51.816668 3163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:53:51.818153 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3163
I20260927 03:53:51.818249 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:53:51.818457 1225 external_mini_cluster.cc:1428] Setting key e7578980253027e7f9a04e2132efbee3
I20260927 03:53:51.819644 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.809608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:53721 (local address 127.1.50.126:43227)
0927 03:53:51.809679 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.809681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.809860 (+ 179us) server_negotiation.cc:407] Connection header received
0927 03:53:51.810418 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.810419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.810433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.810513 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.811351 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.812308 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.813771 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.814092 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.814170 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.816740 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.816742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.816743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.816751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.818095 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.818591 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.818592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.818593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.818659 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.818981 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.818982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.818983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.819141 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.819210 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.819465 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.819552 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:53:51.819880 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.809561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.66:53721)
0927 03:53:51.809726 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.809770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.810322 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.810594 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.810597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.810918 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.811236 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.811242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.812418 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.812419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.813656 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.813659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.814189 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.814189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.814334 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.814832 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.814841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.816625 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.818169 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.818175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.818178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.818490 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.818799 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.818800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.818801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.818907 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.819207 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.819214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.819341 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.819436 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.819672 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260927 03:53:51.820561 3294 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43227
I20260927 03:53:51.820637 3294 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:51.820815 3294 heartbeater.cc:511] Master 127.1.50.126:43227 requested a full tablet report, sending...
I20260927 03:53:51.821556 2951 ts_manager.cc:194] Registered new tserver with Master: c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483)
I20260927 03:53:51.822074 2951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:53721
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:53:51.826759 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.821524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:33229 (local address 127.1.50.126:43227)
0927 03:53:51.821558 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.821560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.821754 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:53:51.821794 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.821794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.821804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.821849 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.822195 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.822887 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.823528 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.823720 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.823777 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.824592 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.824594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.824594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.824600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.825763 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.826085 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.826087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.826087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.826137 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.826290 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.826291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.826292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.826424 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.826479 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.826610 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.826673 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:53:51.826777 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.821561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.66:33229)
0927 03:53:51.821684 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.821692 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.821752 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.821859 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.821860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.822034 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.822091 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.822093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.822962 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.822963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.823464 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.823465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.823777 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.823778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.823847 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.824272 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.824276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.824520 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.825831 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.825831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.825832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.826002 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.826184 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.826185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.826185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.826240 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.826478 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.826479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.826521 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.826581 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.826675 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:53:51.845429 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43227
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:51.926599 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:51.926844 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:51.926883 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:51.928188 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:51.928242 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:51.928274 3303 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:53:51.928293 3303 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:53:51.928329 3303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:53:51.929916 3303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:51.930197 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:51.930444 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:51.935487 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:51.935506 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.935587 3303 server_base.cc:1034] running on GCE node
W20260927 03:53:51.935539 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:51.935928 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:51.936131 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:51.937256 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790481231937247 us; error 27 us; skew 500 ppm
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:51.939589 3303 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:53:51.940023 3303 webserver.cc:460] Webserver started at http://127.1.50.67:41775/ using document root <none> and password file <none>
I20260927 03:53:51.940227 3303 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:51.940268 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:51.940383 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:51.941401 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "4847316c157e49f78eec605f9b6fc3f4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "5e7bbfedd9bd4ba3eb13f546afa1f79f"
server_key_iv: "41c70aed12171053426f9bf5d323a539"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.941723 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "4847316c157e49f78eec605f9b6fc3f4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "5e7bbfedd9bd4ba3eb13f546afa1f79f"
server_key_iv: "41c70aed12171053426f9bf5d323a539"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.942988 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:53:51.943670 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.943806 3303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:53:51.943886 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "4847316c157e49f78eec605f9b6fc3f4"
format_stamp: "Formatted at 2026-09-27 03:53:51 on dist-test-slave-96fg"
server_key: "5e7bbfedd9bd4ba3eb13f546afa1f79f"
server_key_iv: "41c70aed12171053426f9bf5d323a539"
server_key_version: "encryptionkey@0"
I20260927 03:53:51.943946 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:51.963652 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:51.965739 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:51.965873 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:51.966449 3303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:51.966492 3303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.966526 3303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:51.966552 3303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:51.972759 3303 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:43191
I20260927 03:53:51.972836 3433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:43191 every 8 connection(s)
I20260927 03:53:51.973169 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:51.976593 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.966927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:59501 (local address 127.1.50.126:43227)
0927 03:53:51.966970 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.966972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.967236 (+ 264us) server_negotiation.cc:407] Connection header received
0927 03:53:51.967806 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.967808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.967821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.967891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.968865 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.969705 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.971159 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.971381 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.971443 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.974102 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.974105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.974105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.974114 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.975360 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.975737 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.975738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.975738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.975789 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.976070 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.976071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.976071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.976174 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.976224 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.976461 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.976526 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:53:51.976760 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.966937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.67:59501)
0927 03:53:51.967111 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.967148 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.967673 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.968020 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.968025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.968474 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.968775 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.968781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.969787 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.969788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.971077 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.971080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.971479 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.971480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.971647 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.972279 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.972290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.973959 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.975440 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.975446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.975447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.975658 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.975867 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.975868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.975869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.976010 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.976227 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.976231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.976353 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.976435 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.976565 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260927 03:53:51.977186 3434 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43227
I20260927 03:53:51.977290 3434 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:51.977480 3434 heartbeater.cc:511] Master 127.1.50.126:43227 requested a full tablet report, sending...
I20260927 03:53:51.977837 2950 ts_manager.cc:194] Registered new tserver with Master: 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:51.978354 2950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:59501
I20260927 03:53:51.980370 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3303
I20260927 03:53:51.980458 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:53:51.980638 1225 external_mini_cluster.cc:1428] Setting key 745195c7f3976189c139df6c858bddb5
I20260927 03:53:51.981590 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:53:51.983722 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.978218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.1.50.67:41743)
0927 03:53:51.978329 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:51.978339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:51.978399 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:51.978549 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:51.978550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:51.978751 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:51.978815 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.978817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.979411 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.979411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.979938 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:51.979939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.980280 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.980281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:51.980355 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.980733 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:51.980738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:51.981097 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:51.982705 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.982706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.982706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.982907 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.983096 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:51.983097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:51.983097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:51.983161 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:51.983387 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:51.983388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:51.983437 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:53:51.983501 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:53:51.983643 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260927 03:53:51.983692 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.978301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:41743 (local address 127.1.50.126:43227)
0927 03:53:51.978337 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.978339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.978385 (+ 46us) server_negotiation.cc:407] Connection header received
0927 03:53:51.978470 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.978471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.978480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.978543 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.978887 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.979334 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.980016 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.980195 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.980261 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:51.981165 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:51.981167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:51.981167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:51.981173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:51.982612 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.982986 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.982987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.982987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.983038 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:51.983216 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:51.983216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:51.983217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:51.983325 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:51.983388 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:51.983533 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:53:51.983606 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:53:51 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:52.003598 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:51.996928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.1.50.126:43227)
0927 03:53:51.997005 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:51.997007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:51.997015 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:51.997039 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:51.997040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:51.997049 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:51.997095 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:51.997636 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.998185 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.998915 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:51.999116 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:51.999173 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.001390 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:52.001392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:52.001392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:52.001400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:52.002621 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:52.002919 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:52.002920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:52.002921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:52.002970 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:52.003133 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:52.003134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:52.003134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:52.003238 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:52.003319 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.003424 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.003497 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:53:52.005246 2950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:52.006935 2950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:52.013440 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.1.50.66:45483)
0927 03:53:52.009769 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.009774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.009815 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:53:52.009871 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.009873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.009886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.009960 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.010260 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011444 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.012566 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013067 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013131 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.013202 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.013312 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.013348 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":138,"threads_started":1}
I20260927 03:53:52.013432 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:45483 (local address 127.0.0.1:32822)
0927 03:53:52.009762 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.009771 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.009821 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.010002 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.010005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.010011 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.010144 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.010147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011526 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.011527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.012476 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.012488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013143 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.013219 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.013233 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.013292 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.013344 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":31,"thread_start_us":85,"threads_started":1}
I20260927 03:53:52.014103 3451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:43191 (local address 127.0.0.1:42132)
0927 03:53:52.009495 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.009539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.009606 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.010002 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.010005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.010008 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.010173 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.010175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011497 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.011497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.012769 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.012771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013857 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.013915 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.013920 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.013948 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.013986 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":167,"thread_start_us":45,"threads_started":1}
I20260927 03:53:52.014230 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.0.0.1:52864)
0927 03:53:52.010206 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.010215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.010261 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.010469 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.010470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.010472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.010547 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.010548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011379 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.011380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.013105 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.013107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013960 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.014064 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.014068 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.014124 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.014164 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":283,"thread_start_us":84,"threads_started":1}
I20260927 03:53:52.014258 3228 tablet_service.cc:1467] Processing CreateTablet for tablet 434629d54ffc4264824af75e225cd24f (DEFAULT_TABLE table=test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.014446 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.1.50.65:36289)
0927 03:53:52.010099 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.010104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.010314 (+ 210us) server_negotiation.cc:407] Connection header received
0927 03:53:52.010375 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.010378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.010393 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.010461 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.010610 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011290 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013196 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013888 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.013951 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.014005 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.014138 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.014183 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":80,"threads_started":1}
I20260927 03:53:52.014590 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.009581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.1.50.67:43191)
0927 03:53:52.009801 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.009805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.009814 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:53:52.009853 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.009856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.009871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.009960 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.010259 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.011415 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.012860 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.013764 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.014374 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.014439 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.014463 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.014501 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":111,"threads_started":1}
I20260927 03:53:52.014619 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434629d54ffc4264824af75e225cd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.015136 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 434629d54ffc4264824af75e225cd24f (DEFAULT_TABLE table=test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.015132 3368 tablet_service.cc:1467] Processing CreateTablet for tablet 434629d54ffc4264824af75e225cd24f (DEFAULT_TABLE table=test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.015432 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434629d54ffc4264824af75e225cd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.015434 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434629d54ffc4264824af75e225cd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.018229 3457 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Bootstrap starting.
I20260927 03:53:52.018755 3459 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Bootstrap starting.
I20260927 03:53:52.018792 3457 tablet_bootstrap.cc:654] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.018844 3458 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Bootstrap starting.
I20260927 03:53:52.019086 3457 log.cc:824] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:52.019368 3459 tablet_bootstrap.cc:654] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.019647 3458 tablet_bootstrap.cc:654] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.019670 3459 log.cc:824] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:52.019701 3457 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: No bootstrap required, opened a new log
I20260927 03:53:52.019754 3457 ts_tablet_manager.cc:1397] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:52.019959 3458 log.cc:824] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:52.020375 3459 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: No bootstrap required, opened a new log
I20260927 03:53:52.020537 3459 ts_tablet_manager.cc:1397] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:52.020625 3458 tablet_bootstrap.cc:492] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: No bootstrap required, opened a new log
I20260927 03:53:52.020694 3458 ts_tablet_manager.cc:1397] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:52.021675 3457 raft_consensus.cc:348] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.021821 3457 raft_consensus.cc:374] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.021853 3457 raft_consensus.cc:729] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174cd58025343b584d61efe59e220a4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.021986 3457 consensus_queue.cc:260] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.022270 3457 ts_tablet_manager.cc:1428] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:53:52.022389 3294 heartbeater.cc:503] Master 127.1.50.126:43227 was elected leader, sending a full tablet report...
I20260927 03:53:52.022641 3459 raft_consensus.cc:348] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.022641 3458 raft_consensus.cc:348] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.022804 3458 raft_consensus.cc:374] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.022804 3459 raft_consensus.cc:374] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.022843 3459 raft_consensus.cc:729] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.022843 3458 raft_consensus.cc:729] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847316c157e49f78eec605f9b6fc3f4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.022954 3458 consensus_queue.cc:260] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.022966 3459 consensus_queue.cc:260] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.023202 3459 ts_tablet_manager.cc:1428] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:53:52.023202 3458 ts_tablet_manager.cc:1428] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:53:52.023314 3434 heartbeater.cc:503] Master 127.1.50.126:43227 was elected leader, sending a full tablet report...
I20260927 03:53:52.023329 3154 heartbeater.cc:503] Master 127.1.50.126:43227 was elected leader, sending a full tablet report...
W20260927 03:53:52.067270 3295 tablet.cc:2369] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:53:52.070106 3463 raft_consensus.cc:482] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:52.070240 3463 raft_consensus.cc:504] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.070561 3463 leader_election.cc:288] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.073612 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.070859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:44357 (local address 127.1.50.65:36289)
0927 03:53:52.070900 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.070902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.070910 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:52.070940 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.070940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.070952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.071014 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.071203 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.071797 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.072852 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.073287 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.073349 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.073359 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.073495 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.073531 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:53:52.073626 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.070755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.1.50.66:44357)
0927 03:53:52.070802 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.070814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.070860 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.071025 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.071025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.071027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.071105 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.071107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.071868 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.071869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.072779 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.072781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.073354 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.073355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.073436 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.073439 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.073484 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.073524 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260927 03:53:52.073647 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.070828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:52027 (local address 127.1.50.67:43191)
0927 03:53:52.070897 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.070900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.071139 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:53:52.071172 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.071174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.071186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.071248 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.071362 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.071886 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.072938 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.073340 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.073401 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.073412 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.073542 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.073578 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:53:52.073882 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.070760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:43191 (local address 127.1.50.66:52027)
0927 03:53:52.071050 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.071059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.071117 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.071250 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.071250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.071252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.071307 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.071309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.071955 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.071957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.072818 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.072821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.073412 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.073413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.073481 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.073484 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.073527 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.073573 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":122,"thread_start_us":107,"threads_started":1}
I20260927 03:53:52.074011 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434629d54ffc4264824af75e225cd24f" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" is_pre_election: true
I20260927 03:53:52.074194 3108 raft_consensus.cc:2454] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:52.074399 3180 leader_election.cc:302] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.074530 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434629d54ffc4264824af75e225cd24f" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4" is_pre_election: true
I20260927 03:53:52.074666 3388 raft_consensus.cc:2454] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:52.074649 3463 raft_consensus.cc:2781] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:52.074718 3463 raft_consensus.cc:482] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:52.074749 3463 raft_consensus.cc:3037] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.075714 3463 raft_consensus.cc:504] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.075863 3463 leader_election.cc:288] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Requested vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.076056 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434629d54ffc4264824af75e225cd24f" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4"
I20260927 03:53:52.076097 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434629d54ffc4264824af75e225cd24f" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
I20260927 03:53:52.076148 3388 raft_consensus.cc:3037] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.076184 3108 raft_consensus.cc:3037] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.076903 3388 raft_consensus.cc:2454] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:52.076934 3108 raft_consensus.cc:2454] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:52.077076 3180 leader_election.cc:302] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.077203 3463 raft_consensus.cc:2781] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:52.077368 3463 raft_consensus.cc:686] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Becoming Leader. State: Replica: c174cd58025343b584d61efe59e220a4, State: Running, Role: LEADER
I20260927 03:53:52.077459 3463 consensus_queue.cc:237] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.078321 2951 catalog_manager.cc:5495] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c174cd58025343b584d61efe59e220a4 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "c174cd58025343b584d61efe59e220a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:53:52 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481231, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:52.109437 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.102940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.1.50.126:43227)
0927 03:53:52.103000 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.103002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.103100 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:53:52.103191 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.103192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.103203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.103248 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:52.103731 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.104220 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.104884 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.105056 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.105110 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.107123 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:52.107124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:52.107124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:52.107131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:52.108345 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:52.108703 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:52.108704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:52.108704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:52.108754 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:52.108910 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:52.108911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:52.108912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:52.109024 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:52.109077 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.109234 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.109321 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:53:52.109499 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.102832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43227 (local address 127.0.0.1:53692)
0927 03:53:52.103027 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.103037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.103107 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.103258 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.103259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.103575 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:52.103619 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.103622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.104296 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.104297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.104809 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.104811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.105114 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.105115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.105169 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.105666 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:52.105670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:52.107036 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:52.108418 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:52.108425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:52.108428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:52.108609 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:52.108810 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:52.108810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:52.108811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:52.108864 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:52.109076 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:52.109084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:52.109138 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.109205 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.109386 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":73,"thread_start_us":73,"threads_started":1}
I20260927 03:53:52.111711 2951 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:53692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:53:52.111930 2951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:52.113521 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 16fc61d0623648aab9d268d3ed417fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b8fb0ecd1c4f2b879ee498303e7d97]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:52.113524 3228 tablet_service.cc:1467] Processing CreateTablet for tablet 16fc61d0623648aab9d268d3ed417fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b8fb0ecd1c4f2b879ee498303e7d97]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:52.113528 3368 tablet_service.cc:1467] Processing CreateTablet for tablet 16fc61d0623648aab9d268d3ed417fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b8fb0ecd1c4f2b879ee498303e7d97]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:53:52.113679 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fc61d0623648aab9d268d3ed417fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.113682 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fc61d0623648aab9d268d3ed417fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.113682 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fc61d0623648aab9d268d3ed417fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.114771 3458 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: Bootstrap starting.
I20260927 03:53:52.115097 3459 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: Bootstrap starting.
I20260927 03:53:52.115114 3457 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: Bootstrap starting.
I20260927 03:53:52.115258 3458 tablet_bootstrap.cc:654] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.115494 3457 tablet_bootstrap.cc:654] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.115789 3459 tablet_bootstrap.cc:654] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.115931 3458 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: No bootstrap required, opened a new log
I20260927 03:53:52.115981 3458 ts_tablet_manager.cc:1397] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:52.116111 3458 raft_consensus.cc:348] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.116169 3458 raft_consensus.cc:374] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.116187 3458 raft_consensus.cc:729] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847316c157e49f78eec605f9b6fc3f4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.116230 3458 consensus_queue.cc:260] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.116314 3458 ts_tablet_manager.cc:1428] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.116393 3459 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: No bootstrap required, opened a new log
I20260927 03:53:52.116433 3459 ts_tablet_manager.cc:1397] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:52.116397 3464 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.116487 3464 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.116562 3459 raft_consensus.cc:348] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.116636 3459 raft_consensus.cc:374] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.116648 3459 raft_consensus.cc:729] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.116675 3459 consensus_queue.cc:260] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.116715 3457 tablet_bootstrap.cc:492] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: No bootstrap required, opened a new log
I20260927 03:53:52.116755 3459 ts_tablet_manager.cc:1428] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.116762 3457 ts_tablet_manager.cc:1397] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:52.116874 3465 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.116884 3457 raft_consensus.cc:348] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.116941 3457 raft_consensus.cc:374] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.116945 3465 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.116958 3457 raft_consensus.cc:729] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174cd58025343b584d61efe59e220a4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.117003 3457 consensus_queue.cc:260] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.117107 3457 ts_tablet_manager.cc:1428] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.117240 3463 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.117308 3463 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:53:52.163522 3155 tablet.cc:2369] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:53:52.223971 3435 tablet.cc:2369] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:53:52.240835 3465 raft_consensus.cc:482] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:52.240942 3465 raft_consensus.cc:504] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.241314 3465 leader_election.cc:288] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.244050 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.241691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:53629 (local address 127.1.50.66:45483)
0927 03:53:52.241734 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.241736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.241744 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:52.241772 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.241773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.241782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.241826 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.241958 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.242445 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243388 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.243743 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243801 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.243813 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.243931 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.243968 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:53:52.244081 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.241636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:45617 (local address 127.1.50.67:43191)
0927 03:53:52.241685 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.241687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.241736 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:53:52.241825 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.241825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.241833 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.241882 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:53:52.242039 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.242492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243413 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.243761 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243820 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.243830 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.243975 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.244014 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:53:52.244091 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.241511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:45483 (local address 127.1.50.65:53629)
0927 03:53:52.241638 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.241649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.241707 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.241844 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.241844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.241846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.241899 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.241901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.242522 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.242523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.243296 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.243298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.243814 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.243871 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.243875 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.243920 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.243964 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260927 03:53:52.244267 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.241511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:43191 (local address 127.1.50.65:45617)
0927 03:53:52.241661 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.241673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.241754 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.241892 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.241893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.241895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:53:52.241960 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.241962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.242570 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.242571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.243322 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.243324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.243834 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.243835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.243915 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.243918 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.243963 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.244005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260927 03:53:52.244410 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16fc61d0623648aab9d268d3ed417fc5" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c174cd58025343b584d61efe59e220a4" is_pre_election: true
I20260927 03:53:52.244535 3248 raft_consensus.cc:2454] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 0.
I20260927 03:53:52.244516 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16fc61d0623648aab9d268d3ed417fc5" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4" is_pre_election: true
I20260927 03:53:52.244593 3388 raft_consensus.cc:2454] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 0.
I20260927 03:53:52.244724 3039 leader_election.cc:302] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.244843 3465 raft_consensus.cc:2781] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:52.244881 3465 raft_consensus.cc:482] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:52.244901 3465 raft_consensus.cc:3037] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.245529 3465 raft_consensus.cc:504] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.245653 3465 leader_election.cc:288] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 election: Requested vote from peers c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.245795 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16fc61d0623648aab9d268d3ed417fc5" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c174cd58025343b584d61efe59e220a4"
I20260927 03:53:52.245862 3248 raft_consensus.cc:3037] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.246490 3248 raft_consensus.cc:2454] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 1.
I20260927 03:53:52.246667 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16fc61d0623648aab9d268d3ed417fc5" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4"
I20260927 03:53:52.246709 3039 leader_election.cc:302] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.246737 3388 raft_consensus.cc:3037] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.246826 3465 raft_consensus.cc:2781] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:52.246940 3465 raft_consensus.cc:686] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 LEADER]: Becoming Leader. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Running, Role: LEADER
I20260927 03:53:52.247040 3465 consensus_queue.cc:237] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.247437 3388 raft_consensus.cc:2454] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 1.
I20260927 03:53:52.247447 3480 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: TxnStatusTablet state changed. Reason: New leader 416a6c19e54641b4a6b771e9ac0a3ded. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.247532 3480 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:52.247707 3482 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:53:52.247864 3482 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:53:52.247949 2951 catalog_manager.cc:5495] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded reported cstate change: term changed from 0 to 1, leader changed from <none> to 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65). New cstate: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:52.266606 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.263914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.0.0.1:52870)
0927 03:53:52.264005 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.264016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.264065 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.264239 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.264239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.264241 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:52.264294 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.264297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.264917 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.264918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.265533 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.265535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.265929 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.265930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.265998 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.266046 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.266429 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.266441 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.266490 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.266537 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 03:53:52.266620 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.263994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.1.50.65:36289)
0927 03:53:52.264053 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.264056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.264064 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:53:52.264141 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.264142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.264162 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.264218 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:52.264352 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.264842 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.265619 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.265860 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.265914 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.266150 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.266312 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.266437 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.266511 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.266539 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:53:52.268824 3248 raft_consensus.cc:1261] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Refusing update from remote peer 416a6c19e54641b4a6b771e9ac0a3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.268849 3388 raft_consensus.cc:1261] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Refusing update from remote peer 416a6c19e54641b4a6b771e9ac0a3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.269145 3480 consensus_queue.cc:1035] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:52.269273 3465 consensus_queue.cc:1035] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:52.270614 3463 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: TxnStatusTablet state changed. Reason: New leader 416a6c19e54641b4a6b771e9ac0a3ded. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.270649 3464 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: TxnStatusTablet state changed. Reason: New leader 416a6c19e54641b4a6b771e9ac0a3ded. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.270707 3463 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.270740 3464 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.271831 3480 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.271917 3480 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:52.271961 3480 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.272012 3480 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 416a6c19e54641b4a6b771e9ac0a3ded: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:53:52.272238 3463 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.272318 3463 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P c174cd58025343b584d61efe59e220a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.272377 3485 mvcc.cc:205] Tried to move back new op lower bound from 7333811127370498048 to 7333811127284457472. Current Snapshot: MvccSnapshot[applied={T|T < 7333811127370498048}]
I20260927 03:53:52.272455 3486 mvcc.cc:205] Tried to move back new op lower bound from 7333811127370498048 to 7333811127284457472. Current Snapshot: MvccSnapshot[applied={T|T < 7333811127370498048}]
I20260927 03:53:52.272616 3464 tablet_replica.cc:439] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } } }
I20260927 03:53:52.272704 3464 tablet_replica.cc:442] T 16fc61d0623648aab9d268d3ed417fc5 P 4847316c157e49f78eec605f9b6fc3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:53:52.273128 3487 mvcc.cc:205] Tried to move back new op lower bound from 7333811127370498048 to 7333811127284457472. Current Snapshot: MvccSnapshot[applied={T|T < 7333811127370498048}]
I20260927 03:53:52.276899 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.274630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.1.50.66:45483)
0927 03:53:52.274667 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.274669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.274697 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:53:52.274786 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.274787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.274801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.274862 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:52.274981 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.275458 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.276186 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.276408 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.276460 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.276601 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.276706 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.276775 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.276806 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.276832 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:53:52.282150 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.279376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:40537 (local address 127.1.50.65:36289)
0927 03:53:52.279432 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.279434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.279443 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:53:52.279465 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.279466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.279476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.279532 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:52.279631 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.280365 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.281116 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.281433 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.281503 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.281839 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.281879 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.281949 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.282056 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.282076 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:53:52.282186 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.279186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.1.50.66:40537)
0927 03:53:52.279262 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.279273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.279308 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.279539 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.279539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.279541 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:52.279588 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.279590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.280452 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.280453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.281036 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.281039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.281531 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.281532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.281591 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.281607 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.281979 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.281981 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.282053 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.282107 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:53:52.284777 3388 raft_consensus.cc:1261] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.284822 3108 raft_consensus.cc:1261] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.285018 3463 consensus_queue.cc:1035] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:52.285166 3467 consensus_queue.cc:1035] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:52.294013 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.291547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:40065 (local address 127.1.50.66:45483)
0927 03:53:52.291594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:52.291596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:52.291777 (+ 181us) server_negotiation.cc:407] Connection header received
0927 03:53:52.291834 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:52.291835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:52.291845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:52.291894 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:52.292010 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.292523 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.293309 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.293559 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.293621 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.293830 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.293857 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.293892 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:52.293943 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:53:52.293958 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:53:52.294094 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:52.291509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:45483 (local address 127.1.50.65:40065)
0927 03:53:52.291667 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:52.291736 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:52.291790 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:52.291897 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:52.291897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:52.291899 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:52.291954 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.291956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.292593 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.292594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.293211 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:52.293214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:52.293621 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:52.293622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:52.293734 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:52.293750 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:52.293895 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:52.293896 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:53:52.293940 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:53:52.293981 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260927 03:53:52.300326 2951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:52.300441 2951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:52.301227 2951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Sending DeleteTablet for 3 replicas of tablet 434629d54ffc4264824af75e225cd24f
I20260927 03:53:52.301515 3228 tablet_service.cc:1514] Processing DeleteTablet for tablet 434629d54ffc4264824af75e225cd24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:52 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32822
I20260927 03:53:52.301534 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet 434629d54ffc4264824af75e225cd24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:52 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42132
I20260927 03:53:52.301800 3504 tablet_replica.cc:330] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: stopping tablet replica
I20260927 03:53:52.301911 3504 raft_consensus.cc:2229] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:52.301977 3504 raft_consensus.cc:2258] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:52.302083 3088 tablet_service.cc:1514] Processing DeleteTablet for tablet 434629d54ffc4264824af75e225cd24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:52 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:52864
I20260927 03:53:52.302440 3504 ts_tablet_manager.cc:1905] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:52.302639 2951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:52.302803 2951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:52.304092 3228 tablet_service.cc:1467] Processing CreateTablet for tablet 2de16c20adf3460cb69906251ceb98cf (DEFAULT_TABLE table=test-table [id=62d5acaed70d44779319a1985926f063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.304219 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de16c20adf3460cb69906251ceb98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.304306 3504 ts_tablet_manager.cc:1918] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:52.304373 3504 log.cc:1196] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/434629d54ffc4264824af75e225cd24f
I20260927 03:53:52.304612 3504 ts_tablet_manager.cc:1939] T 434629d54ffc4264824af75e225cd24f P 4847316c157e49f78eec605f9b6fc3f4: Deleting consensus metadata
I20260927 03:53:52.305526 3457 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Bootstrap starting.
I20260927 03:53:52.305624 2938 catalog_manager.cc:4841] TS 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191): tablet 434629d54ffc4264824af75e225cd24f (table test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]) successfully deleted
I20260927 03:53:52.305872 3368 tablet_service.cc:1467] Processing CreateTablet for tablet 2de16c20adf3460cb69906251ceb98cf (DEFAULT_TABLE table=test-table [id=62d5acaed70d44779319a1985926f063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.305954 3457 tablet_bootstrap.cc:654] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.305964 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de16c20adf3460cb69906251ceb98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.306542 3457 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: No bootstrap required, opened a new log
I20260927 03:53:52.306581 3457 ts_tablet_manager.cc:1397] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:53:52.306758 3457 raft_consensus.cc:348] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.306819 3457 raft_consensus.cc:374] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.306838 3457 raft_consensus.cc:729] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174cd58025343b584d61efe59e220a4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.306875 3457 consensus_queue.cc:260] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.306954 3457 ts_tablet_manager.cc:1428] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.307119 3458 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Bootstrap starting.
I20260927 03:53:52.307474 3458 tablet_bootstrap.cc:654] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.308208 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 2de16c20adf3460cb69906251ceb98cf (DEFAULT_TABLE table=test-table [id=62d5acaed70d44779319a1985926f063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:52.308327 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de16c20adf3460cb69906251ceb98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:52.309017 3458 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: No bootstrap required, opened a new log
I20260927 03:53:52.309062 3458 ts_tablet_manager.cc:1397] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:52.309191 3458 raft_consensus.cc:348] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.309252 3458 raft_consensus.cc:374] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.309269 3458 raft_consensus.cc:729] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847316c157e49f78eec605f9b6fc3f4, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.309309 3458 consensus_queue.cc:260] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.309394 3458 ts_tablet_manager.cc:1428] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.309530 3459 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Bootstrap starting.
I20260927 03:53:52.309872 3459 tablet_bootstrap.cc:654] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:52.310122 3505 tablet_replica.cc:330] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: stopping tablet replica
I20260927 03:53:52.310256 3505 raft_consensus.cc:2229] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:52.310325 3505 raft_consensus.cc:2258] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:52.310771 3505 ts_tablet_manager.cc:1905] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:52.310940 3459 tablet_bootstrap.cc:492] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: No bootstrap required, opened a new log
I20260927 03:53:52.310974 3459 ts_tablet_manager.cc:1397] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:52.311094 3459 raft_consensus.cc:348] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.311172 3459 raft_consensus.cc:374] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:52.311192 3459 raft_consensus.cc:729] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Initialized, Role: FOLLOWER
I20260927 03:53:52.311233 3459 consensus_queue.cc:260] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.311314 3459 ts_tablet_manager.cc:1428] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:52.312597 3505 ts_tablet_manager.cc:1918] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:52.312666 3505 log.cc:1196] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/434629d54ffc4264824af75e225cd24f
I20260927 03:53:52.312937 3505 ts_tablet_manager.cc:1939] T 434629d54ffc4264824af75e225cd24f P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting consensus metadata
I20260927 03:53:52.313338 2936 catalog_manager.cc:4841] TS 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289): tablet 434629d54ffc4264824af75e225cd24f (table test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]) successfully deleted
I20260927 03:53:52.314146 3503 tablet_replica.cc:330] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: stopping tablet replica
I20260927 03:53:52.314270 3503 raft_consensus.cc:2229] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:53:52.314378 3503 raft_consensus.cc:2258] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:52.314877 3503 ts_tablet_manager.cc:1905] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:52.316985 3503 ts_tablet_manager.cc:1918] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:52.317057 3503 log.cc:1196] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/434629d54ffc4264824af75e225cd24f
I20260927 03:53:52.317270 3503 ts_tablet_manager.cc:1939] T 434629d54ffc4264824af75e225cd24f P c174cd58025343b584d61efe59e220a4: Deleting consensus metadata
I20260927 03:53:52.317888 2935 catalog_manager.cc:4841] TS c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483): tablet 434629d54ffc4264824af75e225cd24f (table test-table [id=7b8ae1b5ccf84bd4aff199b339dfabd8]) successfully deleted
I20260927 03:53:52.431609 3500 raft_consensus.cc:482] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:52.431708 3500 raft_consensus.cc:504] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.431851 3500 leader_election.cc:288] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.432083 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de16c20adf3460cb69906251ceb98cf" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4" is_pre_election: true
I20260927 03:53:52.432083 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de16c20adf3460cb69906251ceb98cf" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" is_pre_election: true
I20260927 03:53:52.432183 3388 raft_consensus.cc:2454] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:52.432184 3108 raft_consensus.cc:2454] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:52.432329 3182 leader_election.cc:302] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4847316c157e49f78eec605f9b6fc3f4, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.432443 3500 raft_consensus.cc:2781] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:52.432502 3500 raft_consensus.cc:482] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:52.432518 3500 raft_consensus.cc:3037] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.433193 3500 raft_consensus.cc:504] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.433324 3500 leader_election.cc:288] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Requested vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:52.433480 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de16c20adf3460cb69906251ceb98cf" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
I20260927 03:53:52.433521 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de16c20adf3460cb69906251ceb98cf" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4"
I20260927 03:53:52.433561 3108 raft_consensus.cc:3037] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.433583 3388 raft_consensus.cc:3037] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:52.434216 3388 raft_consensus.cc:2454] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:52.434259 3108 raft_consensus.cc:2454] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:52.434391 3182 leader_election.cc:302] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4847316c157e49f78eec605f9b6fc3f4, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:52.434496 3500 raft_consensus.cc:2781] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:52.434542 3500 raft_consensus.cc:686] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Becoming Leader. State: Replica: c174cd58025343b584d61efe59e220a4, State: Running, Role: LEADER
I20260927 03:53:52.434593 3500 consensus_queue.cc:237] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:52.435077 2951 catalog_manager.cc:5495] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c174cd58025343b584d61efe59e220a4 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "c174cd58025343b584d61efe59e220a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:52.459619 3388 raft_consensus.cc:1261] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.459618 3108 raft_consensus.cc:1261] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:52.459818 3463 consensus_queue.cc:1035] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:52.459932 3500 consensus_queue.cc:1035] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:53:52.464164 3486 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:53:53.988713 2951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:53.988833 2951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:53.989944 2951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Sending DeleteTablet for 3 replicas of tablet 2de16c20adf3460cb69906251ceb98cf
I20260927 03:53:53.990274 3088 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de16c20adf3460cb69906251ceb98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:53 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:52864
I20260927 03:53:53.990311 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de16c20adf3460cb69906251ceb98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:53 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42132
I20260927 03:53:53.990353 3228 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de16c20adf3460cb69906251ceb98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:53 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32822
I20260927 03:53:53.994891 3534 tablet_replica.cc:330] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: stopping tablet replica
I20260927 03:53:53.994972 3534 raft_consensus.cc:2229] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:53.995018 3534 raft_consensus.cc:2258] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:53.995231 3535 tablet_replica.cc:330] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: stopping tablet replica
I20260927 03:53:53.995281 3534 ts_tablet_manager.cc:1905] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:53.995380 3536 tablet_replica.cc:330] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: stopping tablet replica
I20260927 03:53:53.995517 3536 raft_consensus.cc:2229] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:53:53.995620 3535 raft_consensus.cc:2229] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:53.995699 3535 raft_consensus.cc:2258] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:53.995800 3536 raft_consensus.cc:2258] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:53.996193 3535 ts_tablet_manager.cc:1905] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:53.996465 3536 ts_tablet_manager.cc:1905] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:53.997004 3534 ts_tablet_manager.cc:1918] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:53:53.997087 3534 log.cc:1196] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/2de16c20adf3460cb69906251ceb98cf
I20260927 03:53:53.997363 3534 ts_tablet_manager.cc:1939] T 2de16c20adf3460cb69906251ceb98cf P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting consensus metadata
I20260927 03:53:53.997742 2936 catalog_manager.cc:4841] TS 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289): tablet 2de16c20adf3460cb69906251ceb98cf (table test-table [id=62d5acaed70d44779319a1985926f063]) successfully deleted
I20260927 03:53:53.998147 3536 ts_tablet_manager.cc:1918] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:53:53.998203 3536 log.cc:1196] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/2de16c20adf3460cb69906251ceb98cf
I20260927 03:53:53.998427 3536 ts_tablet_manager.cc:1939] T 2de16c20adf3460cb69906251ceb98cf P c174cd58025343b584d61efe59e220a4: Deleting consensus metadata
I20260927 03:53:53.998808 2935 catalog_manager.cc:4841] TS c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483): tablet 2de16c20adf3460cb69906251ceb98cf (table test-table [id=62d5acaed70d44779319a1985926f063]) successfully deleted
I20260927 03:53:54.001395 3535 ts_tablet_manager.cc:1918] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:53:54.001461 3535 log.cc:1196] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/2de16c20adf3460cb69906251ceb98cf
I20260927 03:53:54.001641 3535 ts_tablet_manager.cc:1939] T 2de16c20adf3460cb69906251ceb98cf P 4847316c157e49f78eec605f9b6fc3f4: Deleting consensus metadata
I20260927 03:53:54.002048 2938 catalog_manager.cc:4841] TS 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191): tablet 2de16c20adf3460cb69906251ceb98cf (table test-table [id=62d5acaed70d44779319a1985926f063]) successfully deleted
Sep 27 03:53:53 dist-test-slave-96fg krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:53:54 dist-test-slave-96fg krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:54.016601 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:54.008525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.1.50.126:43227)
0927 03:53:54.008704 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:54.008706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:54.008876 (+ 170us) server_negotiation.cc:407] Connection header received
0927 03:53:54.008942 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:54.008943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:54.008958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:54.009060 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:54.009713 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.010303 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.011170 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.011435 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.011532 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:54.013960 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:54.013962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:54.013962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:54.013969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:54.015316 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:54.015674 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:54.015674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:54.015674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:54.015760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:54.015953 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:54.015956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:54.015956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:54.016123 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:54.016244 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:54.016423 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 03:53:54.016514 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260927 03:53:54.018356 2951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:54.018551 2951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:54.020254 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c (DEFAULT_TABLE table=test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.020284 3368 tablet_service.cc:1467] Processing CreateTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c (DEFAULT_TABLE table=test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.020393 3228 tablet_service.cc:1467] Processing CreateTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c (DEFAULT_TABLE table=test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.020465 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342a561fd92e42019e7ba6edaf5ccc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.020491 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342a561fd92e42019e7ba6edaf5ccc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.020499 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342a561fd92e42019e7ba6edaf5ccc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.021948 3545 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Bootstrap starting.
I20260927 03:53:54.022205 3547 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Bootstrap starting.
I20260927 03:53:54.022441 3545 tablet_bootstrap.cc:654] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.022603 3547 tablet_bootstrap.cc:654] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.022795 3546 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Bootstrap starting.
I20260927 03:53:54.023144 3545 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: No bootstrap required, opened a new log
I20260927 03:53:54.023213 3546 tablet_bootstrap.cc:654] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.023242 3547 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: No bootstrap required, opened a new log
I20260927 03:53:54.023286 3547 ts_tablet_manager.cc:1397] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:54.023339 3545 ts_tablet_manager.cc:1397] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:54.023399 3547 raft_consensus.cc:348] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.023470 3547 raft_consensus.cc:374] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.023484 3547 raft_consensus.cc:729] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847316c157e49f78eec605f9b6fc3f4, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.023558 3547 consensus_queue.cc:260] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.023619 3545 raft_consensus.cc:348] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.023715 3545 raft_consensus.cc:374] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.023746 3545 raft_consensus.cc:729] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.023739 3547 ts_tablet_manager.cc:1428] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:54.023885 3545 consensus_queue.cc:260] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.023926 3546 tablet_bootstrap.cc:492] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: No bootstrap required, opened a new log
I20260927 03:53:54.024010 3545 ts_tablet_manager.cc:1428] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:53:54.024082 3546 ts_tablet_manager.cc:1397] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:54.024228 3546 raft_consensus.cc:348] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.024313 3546 raft_consensus.cc:374] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.024343 3546 raft_consensus.cc:729] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174cd58025343b584d61efe59e220a4, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.024389 3546 consensus_queue.cc:260] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.024493 3546 ts_tablet_manager.cc:1428] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:54.114039 3513 raft_consensus.cc:482] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:54.114202 3513 raft_consensus.cc:504] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.114440 3513 leader_election.cc:288] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:54.114688 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342a561fd92e42019e7ba6edaf5ccc7c" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4" is_pre_election: true
I20260927 03:53:54.114673 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342a561fd92e42019e7ba6edaf5ccc7c" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" is_pre_election: true
I20260927 03:53:54.114818 3388 raft_consensus.cc:2454] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:54.114836 3108 raft_consensus.cc:2454] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 0.
I20260927 03:53:54.114993 3182 leader_election.cc:302] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4847316c157e49f78eec605f9b6fc3f4, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:54.115159 3513 raft_consensus.cc:2781] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:54.115219 3513 raft_consensus.cc:482] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:54.115239 3513 raft_consensus.cc:3037] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.115957 3513 raft_consensus.cc:504] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.116145 3513 leader_election.cc:288] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Requested vote from peers 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:54.116410 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342a561fd92e42019e7ba6edaf5ccc7c" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4"
I20260927 03:53:54.116425 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342a561fd92e42019e7ba6edaf5ccc7c" candidate_uuid: "c174cd58025343b584d61efe59e220a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416a6c19e54641b4a6b771e9ac0a3ded"
I20260927 03:53:54.116500 3388 raft_consensus.cc:3037] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.116518 3108 raft_consensus.cc:3037] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.117120 3388 raft_consensus.cc:2454] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:54.117301 3108 raft_consensus.cc:2454] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c174cd58025343b584d61efe59e220a4 in term 1.
I20260927 03:53:54.117362 3182 leader_election.cc:302] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4847316c157e49f78eec605f9b6fc3f4, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:54.117456 3513 raft_consensus.cc:2781] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:54.117511 3513 raft_consensus.cc:686] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Becoming Leader. State: Replica: c174cd58025343b584d61efe59e220a4, State: Running, Role: LEADER
I20260927 03:53:54.117564 3513 consensus_queue.cc:237] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.118266 2951 catalog_manager.cc:5495] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c174cd58025343b584d61efe59e220a4 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "c174cd58025343b584d61efe59e220a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:54.145735 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:54.142709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.1.50.66:45483)
0927 03:53:54.143052 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:54.143054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:54.143064 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:53:54.143085 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:54.143085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:54.143097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:54.143171 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:54.143412 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.144042 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.144946 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.145146 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.145210 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:54.145433 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:54.145469 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:54.145534 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:54.145628 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:53:54.145660 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":370,"threads_started":1}
I20260927 03:53:54.148988 3108 raft_consensus.cc:1261] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:54.149309 3388 raft_consensus.cc:1261] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Refusing update from remote peer c174cd58025343b584d61efe59e220a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:54.149605 3513 consensus_queue.cc:1035] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:54.149755 3513 consensus_queue.cc:1035] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:54.160723 2950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:54.160831 2950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:54.161514 2950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Sending DeleteTablet for 3 replicas of tablet 342a561fd92e42019e7ba6edaf5ccc7c
I20260927 03:53:54.161774 3088 tablet_service.cc:1514] Processing DeleteTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:54 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:52864
I20260927 03:53:54.161788 3228 tablet_service.cc:1514] Processing DeleteTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:54 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32822
I20260927 03:53:54.161840 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet 342a561fd92e42019e7ba6edaf5ccc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:54 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42132
I20260927 03:53:54.161882 3534 tablet_replica.cc:330] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: stopping tablet replica
I20260927 03:53:54.161909 3536 tablet_replica.cc:330] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: stopping tablet replica
I20260927 03:53:54.161931 3534 raft_consensus.cc:2229] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:54.161962 3536 raft_consensus.cc:2229] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:53:54.161975 3534 raft_consensus.cc:2258] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:54.161966 3535 tablet_replica.cc:330] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: stopping tablet replica
I20260927 03:53:54.162056 3536 raft_consensus.cc:2258] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:54.162060 3535 raft_consensus.cc:2229] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:54.162103 3535 raft_consensus.cc:2258] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:54.162077 2950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:53:54.162223 2950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:53:54.162580 3536 ts_tablet_manager.cc:1905] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:54.162652 3535 ts_tablet_manager.cc:1905] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:54.162949 3534 ts_tablet_manager.cc:1905] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:54.163599 3088 tablet_service.cc:1467] Processing CreateTablet for tablet 996e0aa4fc5748069a43889921509d1e (DEFAULT_TABLE table=test-table [id=7c56b0530eef48858500d9f627a77c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.163721 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996e0aa4fc5748069a43889921509d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.164338 3535 ts_tablet_manager.cc:1918] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:54.164402 3535 log.cc:1196] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/342a561fd92e42019e7ba6edaf5ccc7c
I20260927 03:53:54.164420 3536 ts_tablet_manager.cc:1918] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:54.164475 3536 log.cc:1196] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/342a561fd92e42019e7ba6edaf5ccc7c
I20260927 03:53:54.164636 3535 ts_tablet_manager.cc:1939] T 342a561fd92e42019e7ba6edaf5ccc7c P 4847316c157e49f78eec605f9b6fc3f4: Deleting consensus metadata
I20260927 03:53:54.164711 3536 ts_tablet_manager.cc:1939] T 342a561fd92e42019e7ba6edaf5ccc7c P c174cd58025343b584d61efe59e220a4: Deleting consensus metadata
I20260927 03:53:54.164832 3534 ts_tablet_manager.cc:1918] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:53:54.164883 3534 log.cc:1196] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/342a561fd92e42019e7ba6edaf5ccc7c
I20260927 03:53:54.165024 3228 tablet_service.cc:1467] Processing CreateTablet for tablet 996e0aa4fc5748069a43889921509d1e (DEFAULT_TABLE table=test-table [id=7c56b0530eef48858500d9f627a77c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.165081 3534 ts_tablet_manager.cc:1939] T 342a561fd92e42019e7ba6edaf5ccc7c P 416a6c19e54641b4a6b771e9ac0a3ded: Deleting consensus metadata
I20260927 03:53:54.165117 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996e0aa4fc5748069a43889921509d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.165542 3545 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded: Bootstrap starting.
I20260927 03:53:54.165668 2936 catalog_manager.cc:4841] TS 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289): tablet 342a561fd92e42019e7ba6edaf5ccc7c (table test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]) successfully deleted
I20260927 03:53:54.165858 2938 catalog_manager.cc:4841] TS 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191): tablet 342a561fd92e42019e7ba6edaf5ccc7c (table test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]) successfully deleted
I20260927 03:53:54.165928 3545 tablet_bootstrap.cc:654] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.166172 3368 tablet_service.cc:1467] Processing CreateTablet for tablet 996e0aa4fc5748069a43889921509d1e (DEFAULT_TABLE table=test-table [id=7c56b0530eef48858500d9f627a77c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:53:54.166298 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996e0aa4fc5748069a43889921509d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:54.166442 3546 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Bootstrap starting.
I20260927 03:53:54.166672 2935 catalog_manager.cc:4841] TS c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483): tablet 342a561fd92e42019e7ba6edaf5ccc7c (table test-table [id=de6d35ac7c354c7ea5558f5a9697a67a]) successfully deleted
I20260927 03:53:54.166831 3546 tablet_bootstrap.cc:654] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.167151 3545 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded: No bootstrap required, opened a new log
I20260927 03:53:54.167202 3545 ts_tablet_manager.cc:1397] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:53:54.167326 3545 raft_consensus.cc:348] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.167389 3545 raft_consensus.cc:374] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.167409 3545 raft_consensus.cc:729] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.167440 3546 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: No bootstrap required, opened a new log
I20260927 03:53:54.167448 3547 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Bootstrap starting.
I20260927 03:53:54.167452 3545 consensus_queue.cc:260] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.167486 3546 ts_tablet_manager.cc:1397] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:54.167536 3545 ts_tablet_manager.cc:1428] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:54.167620 3546 raft_consensus.cc:348] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.167696 3546 raft_consensus.cc:374] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.167714 3546 raft_consensus.cc:729] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174cd58025343b584d61efe59e220a4, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.167758 3546 consensus_queue.cc:260] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.167824 3547 tablet_bootstrap.cc:654] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:54.167866 3546 ts_tablet_manager.cc:1428] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:54.168526 3547 tablet_bootstrap.cc:492] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: No bootstrap required, opened a new log
I20260927 03:53:54.168579 3547 ts_tablet_manager.cc:1397] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:54.168740 3547 raft_consensus.cc:348] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.168809 3547 raft_consensus.cc:374] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:54.168823 3547 raft_consensus.cc:729] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847316c157e49f78eec605f9b6fc3f4, State: Initialized, Role: FOLLOWER
I20260927 03:53:54.168866 3547 consensus_queue.cc:260] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.168958 3547 ts_tablet_manager.cc:1428] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:54.193029 3480 raft_consensus.cc:482] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:53:54.193163 3480 raft_consensus.cc:504] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.193383 3480 leader_election.cc:288] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:54.193607 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996e0aa4fc5748069a43889921509d1e" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c174cd58025343b584d61efe59e220a4" is_pre_election: true
I20260927 03:53:54.193717 3248 raft_consensus.cc:2454] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 0.
I20260927 03:53:54.193717 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996e0aa4fc5748069a43889921509d1e" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4" is_pre_election: true
I20260927 03:53:54.193862 3388 raft_consensus.cc:2454] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 0.
I20260927 03:53:54.193912 3039 leader_election.cc:302] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:54.194038 3480 raft_consensus.cc:2781] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:53:54.194077 3480 raft_consensus.cc:482] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:53:54.194096 3480 raft_consensus.cc:3037] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.194974 3480 raft_consensus.cc:504] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.195108 3480 leader_election.cc:288] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 election: Requested vote from peers c174cd58025343b584d61efe59e220a4 (127.1.50.66:45483), 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191)
I20260927 03:53:54.195312 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996e0aa4fc5748069a43889921509d1e" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c174cd58025343b584d61efe59e220a4"
I20260927 03:53:54.195303 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996e0aa4fc5748069a43889921509d1e" candidate_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4847316c157e49f78eec605f9b6fc3f4"
I20260927 03:53:54.195399 3248 raft_consensus.cc:3037] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.195399 3388 raft_consensus.cc:3037] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:54.196218 3248 raft_consensus.cc:2454] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 1.
I20260927 03:53:54.196218 3388 raft_consensus.cc:2454] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416a6c19e54641b4a6b771e9ac0a3ded in term 1.
I20260927 03:53:54.196451 3039 leader_election.cc:302] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416a6c19e54641b4a6b771e9ac0a3ded, c174cd58025343b584d61efe59e220a4; no voters:
I20260927 03:53:54.196555 3480 raft_consensus.cc:2781] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:54.196660 3480 raft_consensus.cc:686] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [term 1 LEADER]: Becoming Leader. State: Replica: 416a6c19e54641b4a6b771e9ac0a3ded, State: Running, Role: LEADER
I20260927 03:53:54.196755 3480 consensus_queue.cc:237] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } }
I20260927 03:53:54.197378 2949 catalog_manager.cc:5495] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded reported cstate change: term changed from 0 to 1, leader changed from <none> to 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65). New cstate: current_term: 1 leader_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416a6c19e54641b4a6b771e9ac0a3ded" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260927 03:53:54.209895 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:54.207391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.1.50.65:36289)
0927 03:53:54.207582 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:54.207584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:54.207589 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:53:54.207606 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:54.207607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:54.207616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:54.207669 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:54.207865 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.208389 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.209252 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.209444 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.209499 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:54.209650 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:54.209679 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:54.209721 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:54.209806 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:53:54.209824 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
I20260927 03:53:54.215267 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:54.212656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:52943 (local address 127.1.50.65:36289)
0927 03:53:54.212705 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:54.212707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:54.212812 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:53:54.212873 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:54.212874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:54.212883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:54.212929 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:53:54.213153 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.213650 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.214607 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.214839 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.214900 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:54.215027 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:53:54.215056 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:53:54.215093 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:54.215145 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:53:54.215170 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:53:54.215281 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:54.212587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.1.50.65:52943)
0927 03:53:54.212740 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:54.212751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:54.212811 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:54.212966 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:54.212968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:54.212970 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:53:54.213022 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:54.213024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.213784 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.213785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:54.214515 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:54.214518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:54.214913 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:54.214914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:54.214957 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:54.214962 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:53:54.215102 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:53:54.215103 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:53:54.215131 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:53:54.215176 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260927 03:53:54.217494 3248 raft_consensus.cc:1261] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Refusing update from remote peer 416a6c19e54641b4a6b771e9ac0a3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:54.217512 3388 raft_consensus.cc:1261] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Refusing update from remote peer 416a6c19e54641b4a6b771e9ac0a3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:53:54.217725 3516 consensus_queue.cc:1035] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "c174cd58025343b584d61efe59e220a4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:53:54.217844 3480 consensus_queue.cc:1035] T 996e0aa4fc5748069a43889921509d1e P 416a6c19e54641b4a6b771e9ac0a3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "4847316c157e49f78eec605f9b6fc3f4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:53:54.223073 3514 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:53:55.744946 2951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:55.745048 2951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:53:55.745859 2951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd23d002619d44b2a044386193399d60: Sending DeleteTablet for 3 replicas of tablet 996e0aa4fc5748069a43889921509d1e
I20260927 03:53:55.746150 3228 tablet_service.cc:1514] Processing DeleteTablet for tablet 996e0aa4fc5748069a43889921509d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:55 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32822
I20260927 03:53:55.746124 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet 996e0aa4fc5748069a43889921509d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:53:55 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42132
I20260927 03:53:55.746227 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3023
I20260927 03:53:55.750118 3591 tablet_replica.cc:330] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: stopping tablet replica
I20260927 03:53:55.750197 3591 raft_consensus.cc:2229] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:55.750245 3591 raft_consensus.cc:2258] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:55.750348 3590 tablet_replica.cc:330] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: stopping tablet replica
I20260927 03:53:55.750401 3590 raft_consensus.cc:2229] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:53:55.750442 3590 raft_consensus.cc:2258] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:53:55.750638 3591 ts_tablet_manager.cc:1905] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:53:55.750638 3590 ts_tablet_manager.cc:1905] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:53:55.752399 2936 connection.cc:535] client connection to 127.1.50.65:36289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:53:55.752508 2936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:53:55.752521 3591 ts_tablet_manager.cc:1918] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:53:55.752563 3591 log.cc:1196] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/996e0aa4fc5748069a43889921509d1e
I20260927 03:53:55.752574 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3163
I20260927 03:53:55.752763 3591 ts_tablet_manager.cc:1939] T 996e0aa4fc5748069a43889921509d1e P c174cd58025343b584d61efe59e220a4: Deleting consensus metadata
I20260927 03:53:55.753083 3590 ts_tablet_manager.cc:1918] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:53:55.753129 3590 log.cc:1196] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/996e0aa4fc5748069a43889921509d1e
I20260927 03:53:55.753314 3590 ts_tablet_manager.cc:1939] T 996e0aa4fc5748069a43889921509d1e P 4847316c157e49f78eec605f9b6fc3f4: Deleting consensus metadata
I20260927 03:53:55.753796 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:55.753297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36289 (local address 127.0.0.1:52886)
0927 03:53:55.753707 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:55.753757 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":63,"threads_started":1}
I20260927 03:53:55.753902 2938 catalog_manager.cc:4841] TS 4847316c157e49f78eec605f9b6fc3f4 (127.1.50.67:43191): tablet 996e0aa4fc5748069a43889921509d1e (table test-table [id=7c56b0530eef48858500d9f627a77c5f]) successfully deleted
W20260927 03:53:55.753947 2936 catalog_manager.cc:4584] TS 416a6c19e54641b4a6b771e9ac0a3ded (127.1.50.65:36289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 996e0aa4fc5748069a43889921509d1e: Network error: Client connection negotiation failed: client connection to 127.1.50.65:36289: connect: Connection refused (error 111)
I20260927 03:53:55.757946 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3303
I20260927 03:53:55.757984 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:55.757872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:45483 (local address 127.0.0.1:32854)
0927 03:53:55.757922 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:55.757950 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.66:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:53:55.762225 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 2919
2026-09-27T03:53:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5516 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:55 dist-test-slave-96fg krb5kdc[3596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:55 dist-test-slave-96fg krb5kdc[3596](info): set up 2 sockets
Sep 27 03:53:55 dist-test-slave-96fg krb5kdc[3596](info): commencing operation
krb5kdc: starting...
W20260927 03:53:56.717641 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:56 dist-test-slave-96fg krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:56.837240 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46353
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:46353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:56.916893 3609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:56.917145 3609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:56.917176 3609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:56.918449 3609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:56.918500 3609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:56.918520 3609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:56.918537 3609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:56.918552 3609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:56.920210 3609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:46353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:56.920475 3609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:56.920701 3609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:56.925760 3614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:56.925832 3617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:56.925760 3615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:56.926126 3609 server_base.cc:1034] running on GCE node
I20260927 03:53:56.926388 3609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:56.926677 3609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:56.927845 3609 hybrid_clock.cc:648] HybridClock initialized: now 1790481236927812 us; error 48 us; skew 500 ppm
Sep 27 03:53:56 dist-test-slave-96fg krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481236, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:56.930518 3609 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:56.931043 3609 webserver.cc:460] Webserver started at http://127.1.50.126:42429/ using document root <none> and password file <none>
I20260927 03:53:56.931255 3609 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:56.931308 3609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:56.931442 3609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:56.932578 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "3fcba58909b448d98ef141c0ad5cbd80"
format_stamp: "Formatted at 2026-09-27 03:53:56 on dist-test-slave-96fg"
server_key: "2429be1d36dffbf3cb85e620e51cf2a1"
server_key_iv: "ace73a3e74ee8adc8de3be8b39b37405"
server_key_version: "encryptionkey@0"
I20260927 03:53:56.932932 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "3fcba58909b448d98ef141c0ad5cbd80"
format_stamp: "Formatted at 2026-09-27 03:53:56 on dist-test-slave-96fg"
server_key: "2429be1d36dffbf3cb85e620e51cf2a1"
server_key_iv: "ace73a3e74ee8adc8de3be8b39b37405"
server_key_version: "encryptionkey@0"
I20260927 03:53:56.934422 3609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:53:56.935201 3624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:56.935510 3609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:53:56.935616 3609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "3fcba58909b448d98ef141c0ad5cbd80"
format_stamp: "Formatted at 2026-09-27 03:53:56 on dist-test-slave-96fg"
server_key: "2429be1d36dffbf3cb85e620e51cf2a1"
server_key_iv: "ace73a3e74ee8adc8de3be8b39b37405"
server_key_version: "encryptionkey@0"
I20260927 03:53:56.935696 3609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:56.949962 3609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:56.952173 3609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:56.952306 3609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:56.956044 3609 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:46353
I20260927 03:53:56.956090 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:46353 every 8 connection(s)
I20260927 03:53:56.956411 3609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:56.957212 3677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:56.959532 3677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: Bootstrap starting.
I20260927 03:53:56.960127 3677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:56.960399 3677 log.cc:824] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:56.961071 3677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: No bootstrap required, opened a new log
I20260927 03:53:56.961990 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3609
I20260927 03:53:56.962101 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:56.962282 1225 external_mini_cluster.cc:1428] Setting key 0e0394371cf5d1d9e1afcc0acf36d88b
I20260927 03:53:56.963042 3677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } }
I20260927 03:53:56.963188 3677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:56.963218 3677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcba58909b448d98ef141c0ad5cbd80, State: Initialized, Role: FOLLOWER
I20260927 03:53:56.963346 3677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } }
I20260927 03:53:56.963413 3677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:56.963443 3677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:56.963505 3677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:56.964377 3677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } }
I20260927 03:53:56.964479 3677 leader_election.cc:302] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcba58909b448d98ef141c0ad5cbd80; no voters:
I20260927 03:53:56.964628 3677 leader_election.cc:288] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:56.964669 3682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:56.964810 3682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcba58909b448d98ef141c0ad5cbd80, State: Running, Role: LEADER
I20260927 03:53:56.964896 3677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:56.964949 3682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } }
I20260927 03:53:56.965310 3682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fcba58909b448d98ef141c0ad5cbd80. Latest consensus state: current_term: 1 leader_uuid: "3fcba58909b448d98ef141c0ad5cbd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } } }
I20260927 03:53:56.965385 3682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:56.965902 3683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fcba58909b448d98ef141c0ad5cbd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcba58909b448d98ef141c0ad5cbd80" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46353 } } }
I20260927 03:53:56.966157 3683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80 [sys.catalog]: This master's current role is: LEADER
W20260927 03:53:56.966718 3697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:53:56.966773 3697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:53:56.966820 3688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:56.966977 3688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:56.968614 3688 catalog_manager.cc:1347] Generated new cluster ID: e61a9da8888b470582fed5f10f2465ff
I20260927 03:53:56.968667 3688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:53:56 dist-test-slave-96fg krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:56.972519 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:56.962808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.1.50.126:46353)
0927 03:53:56.962991 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:56.963002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:56.963043 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:53:56.963148 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:56.963154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:56.963216 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:56.963382 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:56.964006 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:56.965333 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:56.966457 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:56.967064 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:56.967171 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:56.969555 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:56.969559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:56.969559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:56.969570 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:56.971094 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:56.971410 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:56.971413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:56.971413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:56.971522 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:56.971691 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:56.971692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:56.971692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:56.971932 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:56.972078 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:56.972169 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:53:56.972307 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260927 03:53:56.981083 3688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:56.981837 3688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:56.991852 3688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fcba58909b448d98ef141c0ad5cbd80: Generated new TSK 0
I20260927 03:53:56.992058 3688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:57.023208 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:57.102567 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:57.102782 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:57.102805 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:57.104001 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:57.104048 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:57.104074 3703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:53:57.105572 3703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:57.105815 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:57.106050 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:57.110895 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:57.110875 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.111002 3703 server_base.cc:1034] running on GCE node
W20260927 03:53:57.110888 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.111363 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:57.111604 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:57.112761 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790481237112740 us; error 30 us; skew 500 ppm
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:57.115348 3703 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:53:57.115769 3703 webserver.cc:460] Webserver started at http://127.1.50.65:41263/ using document root <none> and password file <none>
I20260927 03:53:57.115983 3703 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:57.116030 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:57.116154 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:57.117172 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "78b59fcfbfbf444f99a3ca49f0274845"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "eb7135a22d7913ef68d2f45a87910180"
server_key_iv: "1599a31ea777740bbcf2bb9245f168e8"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.117499 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "78b59fcfbfbf444f99a3ca49f0274845"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "eb7135a22d7913ef68d2f45a87910180"
server_key_iv: "1599a31ea777740bbcf2bb9245f168e8"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.118832 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:53:57.119552 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.119732 3703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:53:57.119820 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "78b59fcfbfbf444f99a3ca49f0274845"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "eb7135a22d7913ef68d2f45a87910180"
server_key_iv: "1599a31ea777740bbcf2bb9245f168e8"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.119881 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:57.149479 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:57.151643 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:57.151787 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:57.152009 3703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:57.152307 3703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:57.152340 3703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.152369 3703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:57.152387 3703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.158227 3703 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:40473
I20260927 03:53:57.158275 3831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:40473 every 8 connection(s)
I20260927 03:53:57.158697 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:57.168071 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.159538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:36319 (local address 127.1.50.126:46353)
0927 03:53:57.159589 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:57.159591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:57.159838 (+ 247us) server_negotiation.cc:407] Connection header received
0927 03:53:57.160489 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:57.160491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:57.160504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:57.160545 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:57.161557 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.162154 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.163205 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.163407 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.163462 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.165493 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:57.165494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:57.165494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:57.165503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:57.166955 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.167318 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.167319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.167319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.167368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.167553 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.167553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.167553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.167648 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:57.167693 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:57.167930 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 03:53:57.167994 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:53:57.168084 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3703
I20260927 03:53:57.168169 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:53:57.168329 1225 external_mini_cluster.cc:1428] Setting key c15b1f88075339c542f8de70adbb2baa
I20260927 03:53:57.168329 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.159481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46353 (local address 127.1.50.65:36319)
0927 03:53:57.159712 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:57.159762 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:57.160393 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:57.160624 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:57.160628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:57.161243 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:57.161456 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.161461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.162230 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.162231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.163119 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.163123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.163500 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.163501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.163627 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.164099 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:57.164106 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:57.165417 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:57.167026 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.167033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.167035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.167260 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.167415 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.167416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.167416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.167508 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.167692 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:57.167696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:57.167810 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 03:53:57.167921 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:53:57.168108 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260927 03:53:57.168926 3832 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46353
I20260927 03:53:57.169018 3832 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:57.169210 3832 heartbeater.cc:511] Master 127.1.50.126:46353 requested a full tablet report, sending...
I20260927 03:53:57.169744 3641 ts_manager.cc:194] Registered new tserver with Master: 78b59fcfbfbf444f99a3ca49f0274845 (127.1.50.65:40473)
I20260927 03:53:57.170454 3641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:36319
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:53:57.193312 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:57.271718 3837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:57.271922 3837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:57.271953 3837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:57.273088 3837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:57.273128 3837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:57.273162 3837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:53:57.274566 3837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:57.274796 3837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:57.274999 3837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:57.280004 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:57.280118 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.280126 3837 server_base.cc:1034] running on GCE node
W20260927 03:53:57.280004 3845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.280443 3837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:57.280678 3837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:57.281819 3837 hybrid_clock.cc:648] HybridClock initialized: now 1790481237281808 us; error 33 us; skew 500 ppm
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:57.284192 3837 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:53:57.284597 3837 webserver.cc:460] Webserver started at http://127.1.50.66:41487/ using document root <none> and password file <none>
I20260927 03:53:57.284801 3837 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:57.284849 3837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:57.284967 3837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:57.285986 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "cd10dbcdf62640f8b4a8b7cad785c099"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "ff0c748ce99475cfe6ede36e7e449335"
server_key_iv: "296abfb6eadb60806b4612f8bf72768a"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.286353 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "cd10dbcdf62640f8b4a8b7cad785c099"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "ff0c748ce99475cfe6ede36e7e449335"
server_key_iv: "296abfb6eadb60806b4612f8bf72768a"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.287583 3837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:53:57.288267 3852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.288401 3837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.288480 3837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "cd10dbcdf62640f8b4a8b7cad785c099"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "ff0c748ce99475cfe6ede36e7e449335"
server_key_iv: "296abfb6eadb60806b4612f8bf72768a"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.288545 3837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:57.309291 3837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:57.311206 3837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:57.311349 3837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:57.311563 3837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:57.311864 3837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:57.311898 3837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.311928 3837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:57.311944 3837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.317308 3837 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:34133
I20260927 03:53:57.317359 3965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:34133 every 8 connection(s)
I20260927 03:53:57.317646 3837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:53:57.317941 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3837
I20260927 03:53:57.318051 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:53:57.318225 1225 external_mini_cluster.cc:1428] Setting key d5265ea6c3be5fe5ccc7c944546eb91f
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:53:57.327534 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.318343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:36435 (local address 127.1.50.126:46353)
0927 03:53:57.318400 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:57.318402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:57.318621 (+ 219us) server_negotiation.cc:407] Connection header received
0927 03:53:57.319166 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:57.319167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:57.319179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:57.319243 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:57.320288 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.320803 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.322193 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.322393 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.322446 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.324895 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:57.324897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:57.324897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:57.324904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:57.326251 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.326608 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.326609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.326609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.326666 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.326919 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.326920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.326920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.327028 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:57.327105 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:57.327393 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 03:53:57.327467 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:53:57.327767 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.318341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46353 (local address 127.1.50.66:36435)
0927 03:53:57.318491 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:57.318539 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:57.319073 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:57.319302 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:57.319305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:57.319919 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:57.320199 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.320206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.320886 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.320887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.322118 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.322121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.322469 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.322470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.322633 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.323164 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:57.323174 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:57.324764 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:57.326328 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.326333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.326334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.326527 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.326755 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.326756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.326756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.326866 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.327104 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:57.327111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:57.327238 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:53:57.327342 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:53:57.327466 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260927 03:53:57.328217 3966 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46353
I20260927 03:53:57.328320 3966 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
I20260927 03:53:57.328509 3966 heartbeater.cc:511] Master 127.1.50.126:46353 requested a full tablet report, sending...
I20260927 03:53:57.328969 3641 ts_manager.cc:194] Registered new tserver with Master: cd10dbcdf62640f8b4a8b7cad785c099 (127.1.50.66:34133)
I20260927 03:53:57.329348 3641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:36435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:53:57.345785 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:57.425624 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:57.425844 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:57.425868 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:57.427062 3971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:57.427105 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:57.427131 3971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:53:57.428530 3971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:57.428771 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:57.428990 3971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:57.434130 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.434204 3971 server_base.cc:1034] running on GCE node
W20260927 03:53:57.434180 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:57.434130 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:57.434491 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:57.434710 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:57.435861 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790481237435850 us; error 27 us; skew 500 ppm
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:57.438349 3971 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:53:57.438820 3971 webserver.cc:460] Webserver started at http://127.1.50.67:44107/ using document root <none> and password file <none>
I20260927 03:53:57.439029 3971 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:57.439072 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:57.439184 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:57.440237 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "f413a26b19a24d9587545c817f1f0f67"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "96a6bfe57c2c5d60432101245ba663fd"
server_key_iv: "c54a8baf10a0322aa2b9e861c7648e59"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.440546 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "f413a26b19a24d9587545c817f1f0f67"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "96a6bfe57c2c5d60432101245ba663fd"
server_key_iv: "c54a8baf10a0322aa2b9e861c7648e59"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.441864 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:53:57.442693 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.442860 3971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:53:57.442940 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "f413a26b19a24d9587545c817f1f0f67"
format_stamp: "Formatted at 2026-09-27 03:53:57 on dist-test-slave-96fg"
server_key: "96a6bfe57c2c5d60432101245ba663fd"
server_key_iv: "c54a8baf10a0322aa2b9e861c7648e59"
server_key_version: "encryptionkey@0"
I20260927 03:53:57.443011 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:57.460191 3971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:57.462410 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:57.462528 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:57.462761 3971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:53:57.463049 3971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:53:57.463083 3971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.463104 3971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:53:57.463114 3971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:57.468601 3971 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:37491
I20260927 03:53:57.468662 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:37491 every 8 connection(s)
I20260927 03:53:57.468953 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:53:57.470539 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 3971
I20260927 03:53:57.470623 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:53:57.470774 1225 external_mini_cluster.cc:1428] Setting key bc8c95cf5606774a690b2b0e718c49d7
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:57.477583 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.469606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:45659 (local address 127.1.50.126:46353)
0927 03:53:57.469676 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:57.469678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:57.469850 (+ 172us) server_negotiation.cc:407] Connection header received
0927 03:53:57.470400 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:57.470401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:57.470413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:57.470485 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:57.471573 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.472055 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.473011 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.473193 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.473251 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.475338 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:57.475339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:57.475340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:57.475346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:57.476427 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.476769 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.476771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.476771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.476820 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:57.477015 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:57.477016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:57.477016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:57.477115 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:57.477161 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:57.477437 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 03:53:57.477508 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:53:57.477671 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:57.469586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46353 (local address 127.1.50.67:45659)
0927 03:53:57.469736 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:53:57.469772 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:53:57.470283 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:53:57.470542 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:53:57.470546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:53:57.471241 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:53:57.471503 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.471509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.472115 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.472116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.472944 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:53:57.472946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:53:57.473259 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:53:57.473260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:53:57.473352 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:57.473876 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:53:57.473884 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:53:57.475255 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:53:57.476492 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.476498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.476499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.476699 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.476872 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:53:57.476873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:53:57.476873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:53:57.476969 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:53:57.477162 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:53:57.477166 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:53:57.477293 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:53:57.477380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:53:57.477503 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260927 03:53:57.478107 4100 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46353
I20260927 03:53:57.478204 4100 heartbeater.cc:464] Registering TS with master...
I20260927 03:53:57.478374 4100 heartbeater.cc:511] Master 127.1.50.126:46353 requested a full tablet report, sending...
I20260927 03:53:57.478785 3641 ts_manager.cc:194] Registered new tserver with Master: f413a26b19a24d9587545c817f1f0f67 (127.1.50.67:37491)
I20260927 03:53:57.479143 3641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:45659
I20260927 03:53:57.482357 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:53:57.492924 3681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:53:57.491726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.1.50.126:46353)
0927 03:53:57.491782 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:57.491784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:57.491965 (+ 181us) server_negotiation.cc:407] Connection header received
0927 03:53:57.492008 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:57.492009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:57.492023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:57.492092 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:57.492843 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42618: BlockingRecv error: recv got EOF from 127.0.0.1:42618 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:53:57.493769 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3703
I20260927 03:53:57.498395 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3837
I20260927 03:53:57.502614 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3971
I20260927 03:53:57.506423 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 3609
2026-09-27T03:53:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1743 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[4112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[4112](info): set up 2 sockets
Sep 27 03:53:57 dist-test-slave-96fg krb5kdc[4112](info): commencing operation
krb5kdc: starting...
W20260927 03:53:58.457681 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:58.570403 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39599
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:58.648730 4125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:58.648931 4125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:58.648950 4125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:58.650076 4125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:58.650110 4125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:58.650121 4125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:58.650132 4125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:58.650142 4125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:58.651592 4125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:58.651786 4125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:58.651961 4125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:58.656690 4131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:58.656771 4130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:58.656813 4125 server_base.cc:1034] running on GCE node
W20260927 03:53:58.656771 4133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:58.657151 4125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:58.657431 4125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:58.658608 4125 hybrid_clock.cc:648] HybridClock initialized: now 1790481238658591 us; error 32 us; skew 500 ppm
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:58.661127 4125 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:58.661648 4125 webserver.cc:460] Webserver started at http://127.1.50.126:38713/ using document root <none> and password file <none>
I20260927 03:53:58.661862 4125 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:58.661911 4125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:58.662067 4125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:58.663192 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "35bf0f6e80344057bb0b8ea7bfcca644"
format_stamp: "Formatted at 2026-09-27 03:53:58 on dist-test-slave-96fg"
server_key: "40d95162d8ccb935fbeb6d688021f8ea"
server_key_iv: "92f59759fb0b18f49433f1a1270f4ffc"
server_key_version: "encryptionkey@0"
I20260927 03:53:58.663540 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "35bf0f6e80344057bb0b8ea7bfcca644"
format_stamp: "Formatted at 2026-09-27 03:53:58 on dist-test-slave-96fg"
server_key: "40d95162d8ccb935fbeb6d688021f8ea"
server_key_iv: "92f59759fb0b18f49433f1a1270f4ffc"
server_key_version: "encryptionkey@0"
I20260927 03:53:58.664847 4125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:53:58.665537 4140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:58.665704 4125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:53:58.665784 4125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "35bf0f6e80344057bb0b8ea7bfcca644"
format_stamp: "Formatted at 2026-09-27 03:53:58 on dist-test-slave-96fg"
server_key: "40d95162d8ccb935fbeb6d688021f8ea"
server_key_iv: "92f59759fb0b18f49433f1a1270f4ffc"
server_key_version: "encryptionkey@0"
I20260927 03:53:58.665850 4125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:58.677482 4125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:58.679284 4125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:58.679414 4125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:58.683204 4125 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:39599
I20260927 03:53:58.683243 4192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:39599 every 8 connection(s)
I20260927 03:53:58.683564 4125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:58.684361 4193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:58.684875 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4125
I20260927 03:53:58.684947 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:58.685097 1225 external_mini_cluster.cc:1428] Setting key 6af37b48f2e6931fd1c14742aa0bd2c0
I20260927 03:53:58.687158 4193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: Bootstrap starting.
I20260927 03:53:58.687734 4193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:58.687976 4193 log.cc:824] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:58.688609 4193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: No bootstrap required, opened a new log
I20260927 03:53:58.690683 4193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } }
I20260927 03:53:58.690820 4193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:58.690858 4193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bf0f6e80344057bb0b8ea7bfcca644, State: Initialized, Role: FOLLOWER
I20260927 03:53:58.690982 4193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } }
I20260927 03:53:58.691051 4193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:58.691090 4193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:58.691138 4193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:58.691672 4193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } }
I20260927 03:53:58.691766 4193 leader_election.cc:302] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35bf0f6e80344057bb0b8ea7bfcca644; no voters:
I20260927 03:53:58.691920 4193 leader_election.cc:288] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:58.691962 4198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:58.692170 4198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [term 1 LEADER]: Becoming Leader. State: Replica: 35bf0f6e80344057bb0b8ea7bfcca644, State: Running, Role: LEADER
I20260927 03:53:58.692178 4193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:58.692306 4198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } }
I20260927 03:53:58.692617 4198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bf0f6e80344057bb0b8ea7bfcca644. Latest consensus state: current_term: 1 leader_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } } }
I20260927 03:53:58.692692 4198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:58.692657 4200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bf0f6e80344057bb0b8ea7bfcca644" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39599 } } }
I20260927 03:53:58.692747 4200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644 [sys.catalog]: This master's current role is: LEADER
W20260927 03:53:58.694065 4213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:53:58.694155 4213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:53:58.694231 4202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:58.694413 4202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:53:58.695560 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:58.685576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.1.50.126:39599)
0927 03:53:58.685728 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:58.685736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:58.685760 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:53:58.686183 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:58.686190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:58.686241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:58.686356 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:58.687177 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:58.688188 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:58.688994 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:58.689615 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:58.689740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:58.692032 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:58.692036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:58.692037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:58.692050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:58.693823 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:58.694282 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:58.694285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:58.694285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:58.694387 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:58.694608 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:58.694609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:58.694610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:58.694906 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:58.695058 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:58.695236 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:53:58.695370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260927 03:53:58.696115 4202 catalog_manager.cc:1347] Generated new cluster ID: dff5d79249204bf7972ff006ae181cd4
I20260927 03:53:58.696162 4202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:53:58.696429 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:53:58.727722 4202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:58.728439 4202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:58.730938 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:58.727809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.1.50.126:39599)
0927 03:53:58.727846 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:58.727849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:58.727936 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:53:58.728023 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:58.728024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:58.728033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:58.728076 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:58.728415 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:58.728931 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:58.729588 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:58.729815 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:58.729869 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:58.730494 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:58.730496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:58.730496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:53:58.730695 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:58.730711 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:53:58.730717 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:58.730764 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:58.730822 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:53:58.730862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:53:58.732522 4202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35bf0f6e80344057bb0b8ea7bfcca644: Generated new TSK 0
I20260927 03:53:58.732690 4202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:53:58.736500 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4125
2026-09-27T03:53:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1227 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4226](info): set up 2 sockets
Sep 27 03:53:58 dist-test-slave-96fg krb5kdc[4226](info): commencing operation
krb5kdc: starting...
W20260927 03:53:59.681633 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:53:59 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:53:59.799328 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40959
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:53:59.879848 4239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:53:59.880056 4239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:53:59.880076 4239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:53:59.881206 4239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:53:59.881245 4239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:53:59.881258 4239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:53:59.881269 4239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:53:59.881295 4239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:53:59.882781 4239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:53:59.883005 4239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:53:59.883213 4239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:53:59.888250 4244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:59.888371 4245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:53:59.888511 4247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:53:59.889175 4239 server_base.cc:1034] running on GCE node
I20260927 03:53:59.889421 4239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:53:59.889717 4239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:53:59.890888 4239 hybrid_clock.cc:648] HybridClock initialized: now 1790481239890874 us; error 33 us; skew 500 ppm
Sep 27 03:53:59 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481239, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:53:59.893534 4239 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:53:59.894007 4239 webserver.cc:460] Webserver started at http://127.1.50.126:36429/ using document root <none> and password file <none>
I20260927 03:53:59.894270 4239 fs_manager.cc:359] Metadata directory not provided
I20260927 03:53:59.894316 4239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:53:59.894426 4239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:53:59.895496 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "403941ef33bc4b138a42073431af47bb"
format_stamp: "Formatted at 2026-09-27 03:53:59 on dist-test-slave-96fg"
server_key: "03f49043cecd06affb82a8b0124c228c"
server_key_iv: "6c549c8f3891581526a34352b34f2f4d"
server_key_version: "encryptionkey@0"
I20260927 03:53:59.895828 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "403941ef33bc4b138a42073431af47bb"
format_stamp: "Formatted at 2026-09-27 03:53:59 on dist-test-slave-96fg"
server_key: "03f49043cecd06affb82a8b0124c228c"
server_key_iv: "6c549c8f3891581526a34352b34f2f4d"
server_key_version: "encryptionkey@0"
I20260927 03:53:59.897179 4239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:53:59.898072 4254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:53:59.898432 4239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:53:59.898515 4239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "403941ef33bc4b138a42073431af47bb"
format_stamp: "Formatted at 2026-09-27 03:53:59 on dist-test-slave-96fg"
server_key: "03f49043cecd06affb82a8b0124c228c"
server_key_iv: "6c549c8f3891581526a34352b34f2f4d"
server_key_version: "encryptionkey@0"
I20260927 03:53:59.898599 4239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:53:59.913460 4239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:53:59.915633 4239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:53:59.915758 4239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:53:59.919613 4239 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:40959
I20260927 03:53:59.919656 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:40959 every 8 connection(s)
I20260927 03:53:59.919992 4239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:53:59.920945 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:53:59.923194 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb: Bootstrap starting.
I20260927 03:53:59.923753 4307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb: Neither blocks nor log segments found. Creating new log.
I20260927 03:53:59.923993 4307 log.cc:824] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:53:59.924333 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4239
I20260927 03:53:59.924417 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:53:59.924575 1225 external_mini_cluster.cc:1428] Setting key 29deba69e4e72c85d1a8829a386608a6
I20260927 03:53:59.924733 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb: No bootstrap required, opened a new log
I20260927 03:53:59.926824 4307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } }
I20260927 03:53:59.926992 4307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:53:59.927037 4307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403941ef33bc4b138a42073431af47bb, State: Initialized, Role: FOLLOWER
I20260927 03:53:59.927175 4307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } }
I20260927 03:53:59.927259 4307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:53:59.927299 4307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:53:59.927343 4307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:53:59.928043 4307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } }
I20260927 03:53:59.928155 4307 leader_election.cc:302] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403941ef33bc4b138a42073431af47bb; no voters:
I20260927 03:53:59.928326 4307 leader_election.cc:288] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:53:59.928371 4312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:53:59.928556 4312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [term 1 LEADER]: Becoming Leader. State: Replica: 403941ef33bc4b138a42073431af47bb, State: Running, Role: LEADER
I20260927 03:53:59.928712 4312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } }
I20260927 03:53:59.929165 4313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403941ef33bc4b138a42073431af47bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } } }
I20260927 03:53:59.929270 4313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:59.929175 4314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403941ef33bc4b138a42073431af47bb. Latest consensus state: current_term: 1 leader_uuid: "403941ef33bc4b138a42073431af47bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403941ef33bc4b138a42073431af47bb" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40959 } } }
I20260927 03:53:59.929477 4316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:53:59.929493 4314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [sys.catalog]: This master's current role is: LEADER
I20260927 03:53:59.928563 4307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:53:59.929981 4316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:53:59 dist-test-slave-96fg krb5kdc[4226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:53:59.931905 4316 catalog_manager.cc:1347] Generated new cluster ID: 4d662f7512d4400cbc7903987df3ea3f
I20260927 03:53:59.932583 4316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:53:59.935765 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:53:59.925059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.1.50.126:40959)
0927 03:53:59.925237 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:53:59.925248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:53:59.925283 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:53:59.926090 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:53:59.926100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:53:59.926170 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:53:59.926326 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:53:59.926936 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:59.928528 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:59.929635 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:53:59.930289 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:53:59.930398 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:53:59.932896 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:53:59.932900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:53:59.932901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:53:59.932912 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:53:59.934435 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:59.934744 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:59.934748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:59.934748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:59.934823 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:53:59.934988 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:53:59.934989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:53:59.934989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:53:59.935215 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:53:59.935361 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:53:59.935452 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:53:59.935577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 03:53:59.943665 4316 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:53:59.944377 4316 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:53:59.953735 4316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 403941ef33bc4b138a42073431af47bb: Generated new TSK 0
I20260927 03:53:59.953961 4316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:53:59.990252 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:00.074111 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:00.074324 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:00.074348 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:00.075563 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:00.075608 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:00.075634 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:00.077088 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:00.077315 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:00.077530 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:00.082456 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.082509 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.082485 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:00.082760 4333 server_base.cc:1034] running on GCE node
I20260927 03:54:00.082928 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:00.083122 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:00.084249 4333 hybrid_clock.cc:648] HybridClock initialized: now 1790481240084238 us; error 30 us; skew 500 ppm
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:00.086722 4333 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:00.087121 4333 webserver.cc:460] Webserver started at http://127.1.50.65:38483/ using document root <none> and password file <none>
I20260927 03:54:00.087322 4333 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:00.087373 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:00.087474 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:00.088481 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "2c248a4abeac4276a5fd8585e4401c45"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "6757238e2e24eb90256e44fcec019a43"
server_key_iv: "0715fcbc30b7ad87ea1686caa9714972"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.088783 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "2c248a4abeac4276a5fd8585e4401c45"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "6757238e2e24eb90256e44fcec019a43"
server_key_iv: "0715fcbc30b7ad87ea1686caa9714972"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.090145 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:54:00.090864 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.090991 4333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:00.091070 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "2c248a4abeac4276a5fd8585e4401c45"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "6757238e2e24eb90256e44fcec019a43"
server_key_iv: "0715fcbc30b7ad87ea1686caa9714972"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.091138 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:00.106168 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:00.108186 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:00.108323 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:00.108522 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:00.108815 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:00.108848 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.108877 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:00.108901 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.114238 4333 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:44589
I20260927 03:54:00.114295 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:44589 every 8 connection(s)
I20260927 03:54:00.114581 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:00.115442 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4333
I20260927 03:54:00.115521 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:00.115695 1225 external_mini_cluster.cc:1428] Setting key 4d7d09a4040ec1ba0f446ed6c62bb069
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:54:00.125376 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.115241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:60153 (local address 127.1.50.126:40959)
0927 03:54:00.115307 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:00.115310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:00.115542 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:54:00.116043 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:00.116044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:00.116055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:00.116123 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:00.117060 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.117701 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.119201 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.119420 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.119491 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.122164 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:00.122166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:00.122166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:00.122174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:00.123671 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.124206 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.124207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.124207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.124271 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.124633 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.124635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.124635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.124819 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:00.124888 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:00.125207 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:54:00.125302 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:54:00.125537 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.115209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40959 (local address 127.1.50.65:60153)
0927 03:54:00.115381 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:00.115423 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:00.115948 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:00.116185 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:00.116189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:00.116773 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:00.116963 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.116968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.117789 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.117790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.119060 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.119063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.119510 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.119511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.119671 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.120235 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:00.120245 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:00.122047 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:00.123758 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.123769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.123772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.124036 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.124363 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.124364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.124365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.124499 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.124913 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:00.124920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:00.125065 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 03:54:00.125167 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:54:00.125332 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260927 03:54:00.126157 4462 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40959
I20260927 03:54:00.126258 4462 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
I20260927 03:54:00.126458 4462 heartbeater.cc:511] Master 127.1.50.126:40959 requested a full tablet report, sending...
I20260927 03:54:00.127035 4271 ts_manager.cc:194] Registered new tserver with Master: 2c248a4abeac4276a5fd8585e4401c45 (127.1.50.65:44589)
I20260927 03:54:00.127867 4271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:60153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:00.141422 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:00.221438 4467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:00.221647 4467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:00.221678 4467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:00.222892 4467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:00.222935 4467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:00.222970 4467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:00.224642 4467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:00.224912 4467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:00.225129 4467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:00.230154 4475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.230180 4473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.230242 4472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:00.230506 4467 server_base.cc:1034] running on GCE node
I20260927 03:54:00.230700 4467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:00.230901 4467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:00.232072 4467 hybrid_clock.cc:648] HybridClock initialized: now 1790481240232060 us; error 31 us; skew 500 ppm
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:00.234737 4467 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:00.235205 4467 webserver.cc:460] Webserver started at http://127.1.50.66:36937/ using document root <none> and password file <none>
I20260927 03:54:00.235420 4467 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:00.235466 4467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:00.235592 4467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:00.236640 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "628a9070e24845258f49315a6f0328e2"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "e2ae6368bca40fdd8d92f69dafa600ab"
server_key_iv: "0ace75ad2512bed2c3adf79e010d5a77"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.236967 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "628a9070e24845258f49315a6f0328e2"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "e2ae6368bca40fdd8d92f69dafa600ab"
server_key_iv: "0ace75ad2512bed2c3adf79e010d5a77"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.238282 4467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:54:00.239073 4482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.239235 4467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:00.239323 4467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "628a9070e24845258f49315a6f0328e2"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "e2ae6368bca40fdd8d92f69dafa600ab"
server_key_iv: "0ace75ad2512bed2c3adf79e010d5a77"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.239390 4467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:00.260874 4467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:00.263374 4467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:00.263525 4467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:00.263782 4467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:00.264098 4467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:00.264133 4467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.264163 4467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:00.264179 4467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.269606 4467 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:38711
I20260927 03:54:00.269660 4595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:38711 every 8 connection(s)
I20260927 03:54:00.269959 4467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:00.276497 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4467
I20260927 03:54:00.276599 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:00.276853 1225 external_mini_cluster.cc:1428] Setting key c8844942968e25f7a7b8dcb7858c2a81
I20260927 03:54:00.279727 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.270796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:34935 (local address 127.1.50.126:40959)
0927 03:54:00.270856 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:00.270858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:00.271018 (+ 160us) server_negotiation.cc:407] Connection header received
0927 03:54:00.271517 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:00.271518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:00.271530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:00.271577 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:00.272514 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.273009 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.274009 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.274231 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.274293 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.276544 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:00.276546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:00.276547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:00.276556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:00.278244 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.278650 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.278652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.278652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.278713 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.279084 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.279085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.279085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.279207 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:00.279261 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:00.279582 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 03:54:00.279660 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:54:00.279994 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.270686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40959 (local address 127.1.50.66:34935)
0927 03:54:00.270890 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:00.270937 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:00.271424 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:00.271651 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:00.271655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:00.272217 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:00.272423 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.272428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.273092 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.273093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.273928 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.273930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.274307 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.274308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.274430 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.274954 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:00.274963 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:00.276433 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:00.278344 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.278351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.278353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.278580 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.278785 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.278785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.278786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.278960 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.279265 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:00.279271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:00.279445 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 03:54:00.279561 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:54:00.279734 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260927 03:54:00.280480 4596 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40959
I20260927 03:54:00.280581 4596 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:00.280793 4596 heartbeater.cc:511] Master 127.1.50.126:40959 requested a full tablet report, sending...
I20260927 03:54:00.281190 4271 ts_manager.cc:194] Registered new tserver with Master: 628a9070e24845258f49315a6f0328e2 (127.1.50.66:38711)
I20260927 03:54:00.281729 4271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:34935
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:00.305162 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:00.385169 4601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:00.385381 4601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:00.385414 4601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:00.386639 4601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:00.386691 4601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:00.386727 4601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:00.388166 4601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:00.388410 4601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:00.388614 4601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:00.393356 4609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.393380 4607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:00.393380 4606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:00.393718 4601 server_base.cc:1034] running on GCE node
I20260927 03:54:00.393903 4601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:00.394169 4601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:00.395340 4601 hybrid_clock.cc:648] HybridClock initialized: now 1790481240395326 us; error 20 us; skew 500 ppm
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:00.397686 4601 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:00.398172 4601 webserver.cc:460] Webserver started at http://127.1.50.67:41431/ using document root <none> and password file <none>
I20260927 03:54:00.398365 4601 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:00.398404 4601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:00.398502 4601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:00.399538 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "e894e66d203445ec90642e4eda2cca5b"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "467ae127dedd33c2e32951a178552429"
server_key_iv: "b12f9a44be2cb97d21709cc4ae2e9577"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.399828 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "e894e66d203445ec90642e4eda2cca5b"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "467ae127dedd33c2e32951a178552429"
server_key_iv: "b12f9a44be2cb97d21709cc4ae2e9577"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.400997 4601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:00.401755 4616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.401929 4601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:00.402009 4601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "e894e66d203445ec90642e4eda2cca5b"
format_stamp: "Formatted at 2026-09-27 03:54:00 on dist-test-slave-96fg"
server_key: "467ae127dedd33c2e32951a178552429"
server_key_iv: "b12f9a44be2cb97d21709cc4ae2e9577"
server_key_version: "encryptionkey@0"
I20260927 03:54:00.402105 4601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:00.414961 4601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:00.416869 4601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:00.416997 4601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:00.417193 4601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:00.417467 4601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:00.417500 4601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.417528 4601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:00.417554 4601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:00.422917 4601 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40227
I20260927 03:54:00.422973 4729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40227 every 8 connection(s)
I20260927 03:54:00.423331 4601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:00.429852 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4601
I20260927 03:54:00.429942 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:00.430220 1225 external_mini_cluster.cc:1428] Setting key 6c50cb0df4f719e8c9037b8b527f0e03
I20260927 03:54:00.432518 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.424073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:48585 (local address 127.1.50.126:40959)
0927 03:54:00.424235 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:00.424237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:00.424393 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:54:00.424971 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:00.424973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:00.424986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:00.425036 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:00.425970 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.426559 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.427578 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.427781 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.427852 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.430061 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:00.430063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:00.430064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:00.430072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:00.431479 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.431825 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.431826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.431826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.431876 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.432049 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.432049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.432050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.432159 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:00.432205 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:00.432402 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 03:54:00.432457 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 03:54:00.432720 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.424038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40959 (local address 127.1.50.67:48585)
0927 03:54:00.424262 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:00.424308 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:00.424864 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:00.425119 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:00.425123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:00.425680 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:00.425876 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.425880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.426654 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.426655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.427498 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:00.427500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.427868 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.427869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:00.427969 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.428413 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:00.428421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:00.429921 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:00.431542 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.431547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.431549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.431766 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.431922 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:00.431922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:00.431922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:00.432008 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:00.432202 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:00.432205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:00.432313 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 03:54:00.432390 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:54:00.432503 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260927 03:54:00.433158 4730 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40959
I20260927 03:54:00.433247 4730 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:00.433411 4730 heartbeater.cc:511] Master 127.1.50.126:40959 requested a full tablet report, sending...
I20260927 03:54:00.433748 4271 ts_manager.cc:194] Registered new tserver with Master: e894e66d203445ec90642e4eda2cca5b (127.1.50.67:40227)
I20260927 03:54:00.434160 4271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:48585
I20260927 03:54:00.441874 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:00.466279 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:00.456229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.1.50.126:40959)
0927 03:54:00.456293 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:00.456295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:00.456384 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:54:00.456451 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:00.456452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:00.456462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:00.456509 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:00.457061 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.457565 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.458302 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:00.458503 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:00.458554 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:00.464146 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:00.464147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:00.464148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:00.464155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:00.465337 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.465652 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.465653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.465653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.465701 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:00.465827 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:00.465828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:00.465828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:00.465962 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:00.466066 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:00.466146 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:54:00.466213 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 03:54:00.466487 4271 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54360
I20260927 03:54:00.466903 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4333
I20260927 03:54:00.470813 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4467
I20260927 03:54:00.474347 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4601
I20260927 03:54:00.478175 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4239
2026-09-27T03:54:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1743 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4742](info): set up 2 sockets
Sep 27 03:54:00 dist-test-slave-96fg krb5kdc[4742](info): commencing operation
krb5kdc: starting...
W20260927 03:54:01.429843 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:01.545161 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:42523
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:01.623704 4755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:01.623930 4755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:01.623953 4755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:01.625113 4755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:01.625152 4755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:01.625164 4755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:01.625176 4755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:01.625187 4755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:01.626739 4755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:01.626947 4755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:01.627126 4755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:01.631735 4761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:01.631745 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:01.631783 4760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:01.631964 4755 server_base.cc:1034] running on GCE node
I20260927 03:54:01.632259 4755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:01.632539 4755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:01.633705 4755 hybrid_clock.cc:648] HybridClock initialized: now 1790481241633689 us; error 31 us; skew 500 ppm
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:01.640103 4755 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:01.640623 4755 webserver.cc:460] Webserver started at http://127.1.50.126:36691/ using document root <none> and password file <none>
I20260927 03:54:01.640841 4755 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:01.640887 4755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:01.641008 4755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:01.642208 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "fc31325f006445f39da1be704fbbdd7c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "2be00d602e7bc1de3d1821690de6722a"
server_key_iv: "3190c79da74509f30d18a047838c5128"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.642572 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "fc31325f006445f39da1be704fbbdd7c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "2be00d602e7bc1de3d1821690de6722a"
server_key_iv: "3190c79da74509f30d18a047838c5128"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.643996 4755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:01.644897 4770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:01.645071 4755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:01.645154 4755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "fc31325f006445f39da1be704fbbdd7c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "2be00d602e7bc1de3d1821690de6722a"
server_key_iv: "3190c79da74509f30d18a047838c5128"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.645231 4755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:01.669992 4755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:01.672183 4755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:01.672302 4755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:01.676117 4755 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:42523
I20260927 03:54:01.676424 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:42523 every 8 connection(s)
I20260927 03:54:01.676470 4755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:01.677139 4823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:01.679467 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c: Bootstrap starting.
I20260927 03:54:01.680027 4823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:01.680121 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4755
I20260927 03:54:01.680219 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:01.680343 4823 log.cc:824] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:01.680392 1225 external_mini_cluster.cc:1428] Setting key 01ca274a0451ebf417320b4327cc5800
I20260927 03:54:01.681311 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c: No bootstrap required, opened a new log
I20260927 03:54:01.683094 4823 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } }
I20260927 03:54:01.683234 4823 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:01.683266 4823 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc31325f006445f39da1be704fbbdd7c, State: Initialized, Role: FOLLOWER
I20260927 03:54:01.683395 4823 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } }
I20260927 03:54:01.683466 4823 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:01.683507 4823 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:01.683552 4823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:01.684055 4823 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } }
I20260927 03:54:01.684156 4823 leader_election.cc:302] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc31325f006445f39da1be704fbbdd7c; no voters:
I20260927 03:54:01.684322 4823 leader_election.cc:288] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:01.684384 4828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:01.684559 4823 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:01.684626 4828 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [term 1 LEADER]: Becoming Leader. State: Replica: fc31325f006445f39da1be704fbbdd7c, State: Running, Role: LEADER
I20260927 03:54:01.684795 4828 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } }
I20260927 03:54:01.685206 4829 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc31325f006445f39da1be704fbbdd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } } }
I20260927 03:54:01.685290 4829 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:01.685428 4830 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc31325f006445f39da1be704fbbdd7c. Latest consensus state: current_term: 1 leader_uuid: "fc31325f006445f39da1be704fbbdd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc31325f006445f39da1be704fbbdd7c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 42523 } } }
I20260927 03:54:01.685474 4830 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:01.685851 4837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:01.686429 4837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:01.688930 4837 catalog_manager.cc:1347] Generated new cluster ID: 0d41a119c68d42af8a32e36acc7ba570
I20260927 03:54:01.688985 4837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:01.692621 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:01.680901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.1.50.126:42523)
0927 03:54:01.681057 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:01.681065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:01.681093 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:54:01.681593 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:01.681599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:01.681651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:01.681772 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:01.682571 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.684020 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.685115 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.685720 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.685831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:01.688972 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:01.688976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:01.688977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:01.688989 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:01.690942 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:01.691360 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:01.691363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:01.691363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:01.691475 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:01.691664 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:01.691665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:01.691666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:01.691987 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:01.692159 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:01.692248 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 03:54:01.692404 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260927 03:54:01.707281 4837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:01.707846 4837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:01.712646 4837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc31325f006445f39da1be704fbbdd7c: Generated new TSK 0
I20260927 03:54:01.712833 4837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:54:01.737155 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:01.819056 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:01.819257 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:01.819278 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:01.820391 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:01.820432 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:01.820456 4849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:01.821990 4849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:01.822247 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:01.822449 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:54:01.827333 4849 server_base.cc:1034] running on GCE node
W20260927 03:54:01.827303 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:01.827308 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:01.827308 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:01.827787 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:01.827994 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:01.829180 4849 hybrid_clock.cc:648] HybridClock initialized: now 1790481241829163 us; error 33 us; skew 500 ppm
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:01.831650 4849 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:01.832078 4849 webserver.cc:460] Webserver started at http://127.1.50.65:33847/ using document root <none> and password file <none>
I20260927 03:54:01.832269 4849 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:01.832309 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:01.832394 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:01.833412 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "a7c6bc0360544229ab1458947992c1c6"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "a45eff00856ed42095e33052591448fe"
server_key_iv: "4e3f0291ea5ed7c1fd70639eb442c745"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.833695 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "a7c6bc0360544229ab1458947992c1c6"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "a45eff00856ed42095e33052591448fe"
server_key_iv: "4e3f0291ea5ed7c1fd70639eb442c745"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.834892 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:01.835554 4864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:01.835736 4849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:01.835811 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "a7c6bc0360544229ab1458947992c1c6"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "a45eff00856ed42095e33052591448fe"
server_key_iv: "4e3f0291ea5ed7c1fd70639eb442c745"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.835880 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:01.849839 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:01.851868 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:01.851989 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:01.852216 4849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:01.852499 4849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:01.852531 4849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:01.852552 4849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:01.852576 4849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:01.857865 4849 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:42883
I20260927 03:54:01.857918 4977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:42883 every 8 connection(s)
I20260927 03:54:01.858263 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:01.861794 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4849
I20260927 03:54:01.861893 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:01.862080 1225 external_mini_cluster.cc:1428] Setting key 8e74d52aaf44fe0abfc91a78733e62d4
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:01.867926 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:01.859031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37423 (local address 127.1.50.126:42523)
0927 03:54:01.859074 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:01.859077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:01.859292 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:54:01.860242 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:01.860243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:01.860256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:01.860310 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:01.860941 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.861455 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.862637 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.862840 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.862902 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:01.865164 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:01.865166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:01.865166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:01.865174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:01.866636 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:01.867030 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:01.867031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:01.867032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:01.867089 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:01.867305 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:01.867306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:01.867306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:01.867448 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:01.867517 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:01.867742 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 03:54:01.867827 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:54:01.868044 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:01.859019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:42523 (local address 127.1.50.65:37423)
0927 03:54:01.859167 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:01.859203 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:01.860128 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:01.860383 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:01.860388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:01.860654 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:01.860853 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:01.860858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.861538 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.861539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:01.862562 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:01.862564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:01.862926 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:01.862928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:01.863067 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:01.863548 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:01.863556 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:01.865073 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:01.866740 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:01.866746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:01.866748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:01.866962 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:01.867150 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:01.867150 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:01.867150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:01.867248 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:01.867504 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:01.867507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:01.867620 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 03:54:01.867736 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:54:01.867863 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260927 03:54:01.868525 4978 heartbeater.cc:347] Connected to a master server at 127.1.50.126:42523
I20260927 03:54:01.868618 4978 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:01.868785 4978 heartbeater.cc:511] Master 127.1.50.126:42523 requested a full tablet report, sending...
I20260927 03:54:01.869315 4787 ts_manager.cc:194] Registered new tserver with Master: a7c6bc0360544229ab1458947992c1c6 (127.1.50.65:42883)
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:54:01.870240 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:37423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:01.889407 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:01.969786 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:01.970003 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:01.970055 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:01.971243 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:01.971285 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:01.971320 4983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:01.972859 4983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:01.973093 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:01.973310 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:01.978622 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:01.978637 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:01.978694 4983 server_base.cc:1034] running on GCE node
W20260927 03:54:01.978637 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:01.979107 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:01.979341 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:01.980476 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790481241980459 us; error 37 us; skew 500 ppm
Sep 27 03:54:01 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:01.983094 4983 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:01.983546 4983 webserver.cc:460] Webserver started at http://127.1.50.66:40853/ using document root <none> and password file <none>
I20260927 03:54:01.983747 4983 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:01.983800 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:01.983915 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:01.985045 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "21bf485bc32d4453988573a602201a4c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "155d06c6b4ef826d0b8c16e504f04b3b"
server_key_iv: "cb308da913aa646eac5402724456895d"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.985373 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "21bf485bc32d4453988573a602201a4c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "155d06c6b4ef826d0b8c16e504f04b3b"
server_key_iv: "cb308da913aa646eac5402724456895d"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.986734 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:01.987530 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:01.987680 4983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:01.987757 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "21bf485bc32d4453988573a602201a4c"
format_stamp: "Formatted at 2026-09-27 03:54:01 on dist-test-slave-96fg"
server_key: "155d06c6b4ef826d0b8c16e504f04b3b"
server_key_iv: "cb308da913aa646eac5402724456895d"
server_key_version: "encryptionkey@0"
I20260927 03:54:01.987849 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:01.999137 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:02.001050 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:02.001170 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:02.001431 4983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:02.001741 4983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:02.001781 4983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:02.001806 4983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:02.001829 4983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:02.007285 4983 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:33919
I20260927 03:54:02.007326 5111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:33919 every 8 connection(s)
I20260927 03:54:02.007614 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481241, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:02.014283 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 4983
I20260927 03:54:02.014381 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:02.014550 1225 external_mini_cluster.cc:1428] Setting key 3f772cec9ec5a84721a63ccf2eda6111
I20260927 03:54:02.016882 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.008328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:46799 (local address 127.1.50.126:42523)
0927 03:54:02.008394 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:02.008397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:02.008554 (+ 157us) server_negotiation.cc:407] Connection header received
0927 03:54:02.009525 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:02.009526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:02.009539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:02.009590 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:02.010255 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.010753 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.011722 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.011918 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.011974 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.014121 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:02.014123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:02.014123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:02.014131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:02.015604 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.015982 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.015983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.015984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.016040 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.016235 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.016236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.016236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.016354 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:02.016411 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:02.016710 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 03:54:02.016791 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:54:02.017066 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.008265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:42523 (local address 127.1.50.66:46799)
0927 03:54:02.008435 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:02.008475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:02.009416 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:02.009662 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:02.009666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:02.009930 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:02.010165 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:02.010169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.010831 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.010832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:02.011638 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:02.011640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.011983 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.011984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:02.012105 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.012578 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:02.012586 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:02.014003 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:02.015684 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:02.015690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:02.015692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:02.015920 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:02.016092 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:02.016092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:02.016092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:02.016187 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:02.016411 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:02.016417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:02.016543 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 03:54:02.016670 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 03:54:02.016846 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260927 03:54:02.017567 5112 heartbeater.cc:347] Connected to a master server at 127.1.50.126:42523
I20260927 03:54:02.017656 5112 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:02.017822 5112 heartbeater.cc:511] Master 127.1.50.126:42523 requested a full tablet report, sending...
I20260927 03:54:02.018196 4787 ts_manager.cc:194] Registered new tserver with Master: 21bf485bc32d4453988573a602201a4c (127.1.50.66:33919)
I20260927 03:54:02.018610 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:46799
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:02.041934 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:02.120884 5117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:02.121101 5117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:02.121133 5117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:02.122337 5117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:02.122380 5117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:02.122416 5117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:02.123986 5117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:02.124233 5117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:02.124445 5117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:02.129323 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:02.129333 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:02.129484 5117 server_base.cc:1034] running on GCE node
W20260927 03:54:02.129405 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:02.129788 5117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:02.130002 5117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:02.131178 5117 hybrid_clock.cc:648] HybridClock initialized: now 1790481242131161 us; error 32 us; skew 500 ppm
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481242, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:02.133677 5117 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:02.134100 5117 webserver.cc:460] Webserver started at http://127.1.50.67:36303/ using document root <none> and password file <none>
I20260927 03:54:02.134297 5117 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:02.134338 5117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:02.134428 5117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:02.135412 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "86f4682c567f4e35bd6aa389f1b96ce7"
format_stamp: "Formatted at 2026-09-27 03:54:02 on dist-test-slave-96fg"
server_key: "61dc226e6faf28ba8b5d37407f213861"
server_key_iv: "6c74fa2c9c482097a69469eb4f3ac5b3"
server_key_version: "encryptionkey@0"
I20260927 03:54:02.135692 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "86f4682c567f4e35bd6aa389f1b96ce7"
format_stamp: "Formatted at 2026-09-27 03:54:02 on dist-test-slave-96fg"
server_key: "61dc226e6faf28ba8b5d37407f213861"
server_key_iv: "6c74fa2c9c482097a69469eb4f3ac5b3"
server_key_version: "encryptionkey@0"
I20260927 03:54:02.136942 5117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:02.137754 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:02.137969 5117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:02.138078 5117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "86f4682c567f4e35bd6aa389f1b96ce7"
format_stamp: "Formatted at 2026-09-27 03:54:02 on dist-test-slave-96fg"
server_key: "61dc226e6faf28ba8b5d37407f213861"
server_key_iv: "6c74fa2c9c482097a69469eb4f3ac5b3"
server_key_version: "encryptionkey@0"
I20260927 03:54:02.138145 5117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:02.147346 5117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:02.149493 5117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:02.149623 5117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:02.149837 5117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:02.150161 5117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:02.150197 5117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:02.150226 5117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:02.150249 5117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:02.155491 5117 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:39389
I20260927 03:54:02.155556 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:39389 every 8 connection(s)
I20260927 03:54:02.155843 5117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:02.156463 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5117
I20260927 03:54:02.156565 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:02.156764 1225 external_mini_cluster.cc:1428] Setting key 4bf6084445850290a1771d6a550b124b
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481242, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:02.164638 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.156527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:42763 (local address 127.1.50.126:42523)
0927 03:54:02.156723 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:02.156725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:02.156759 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:54:02.157812 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:02.157813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:02.157824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:02.157876 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:02.158616 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.159113 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.160066 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.160257 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.160311 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.162360 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:02.162361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:02.162362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:02.162369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:02.163422 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.163790 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.163791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.163792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.163841 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.164053 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.164053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.164054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.164160 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:02.164215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:02.164469 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 03:54:02.164544 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260927 03:54:02.164814 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.156485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:42523 (local address 127.1.50.67:42763)
0927 03:54:02.156646 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:02.156687 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:02.157718 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:02.157953 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:02.157958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:02.158336 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:02.158537 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:02.158543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.159197 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.159198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:02.159988 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:02.159990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.160322 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.160323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:02.160444 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.160917 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:02.160924 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:02.162275 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:02.163492 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:02.163497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:02.163500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:02.163723 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:02.163894 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:02.163895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:02.163895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:02.164000 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:02.164217 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:02.164223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:02.164357 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 03:54:02.164444 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:54:02.164598 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260927 03:54:02.165192 5246 heartbeater.cc:347] Connected to a master server at 127.1.50.126:42523
I20260927 03:54:02.165282 5246 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:02.165457 5246 heartbeater.cc:511] Master 127.1.50.126:42523 requested a full tablet report, sending...
I20260927 03:54:02.165786 4787 ts_manager.cc:194] Registered new tserver with Master: 86f4682c567f4e35bd6aa389f1b96ce7 (127.1.50.67:39389)
I20260927 03:54:02.166190 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:42763
I20260927 03:54:02.168527 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:54:02.192209 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.183823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.1.50.65:42883)
0927 03:54:02.183967 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:02.183973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:02.183986 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:54:02.184015 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:02.184017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:02.184030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:02.184107 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:02.184739 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.185566 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.186351 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.186702 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.186763 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.189223 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:02.189233 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:02.189234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:02.189241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:02.190967 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.191320 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.191323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.191324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.191404 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.191616 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.191617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.191618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.191772 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:02.191908 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:02.191980 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:54:02.192059 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:02.200388 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:02.193435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.1.50.126:42523)
0927 03:54:02.193483 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:02.193485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:02.193518 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:54:02.193565 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:02.193566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:02.193574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:02.193621 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:02.194139 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.194628 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.195369 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:02.195584 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:02.195645 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:02.198169 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:02.198170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:02.198171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:02.198178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:02.199361 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.199679 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.199680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.199680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.199730 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:02.199931 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:02.199931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:02.199931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:02.200070 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:02.200145 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:02.200251 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:54:02.200319 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 03:54:02.200613 4787 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37814
I20260927 03:54:02.200930 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4849
I20260927 03:54:02.205483 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4983
I20260927 03:54:02.209213 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5117
I20260927 03:54:02.212654 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 4755
2026-09-27T03:54:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1734 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[5258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[5258](info): set up 2 sockets
Sep 27 03:54:02 dist-test-slave-96fg krb5kdc[5258](info): commencing operation
krb5kdc: starting...
W20260927 03:54:03.165776 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:03.284994 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35275
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:03.363469 5271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:03.363670 5271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:03.363692 5271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:03.364773 5271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:03.364804 5271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:03.364815 5271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:03.364826 5271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:03.364836 5271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:03.366268 5271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:03.366456 5271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:03.366637 5271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:03.371388 5276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.371398 5279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.371457 5271 server_base.cc:1034] running on GCE node
W20260927 03:54:03.371402 5277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.371879 5271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:03.372196 5271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:03.373335 5271 hybrid_clock.cc:648] HybridClock initialized: now 1790481243373322 us; error 27 us; skew 500 ppm
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:03.375630 5271 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:03.375917 5271 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:03.375959 5271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:03.376055 5271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:03.377051 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "d1defa8a45cd4634af945f1887ada45d"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "d45e48b747432c4b126362926f7de1c6"
server_key_iv: "42ba2b286d9fbae9161e615955671d41"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.377332 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "d1defa8a45cd4634af945f1887ada45d"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "d45e48b747432c4b126362926f7de1c6"
server_key_iv: "42ba2b286d9fbae9161e615955671d41"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.378492 5271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:54:03.379110 5285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.379257 5271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:03.379333 5271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "d1defa8a45cd4634af945f1887ada45d"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "d45e48b747432c4b126362926f7de1c6"
server_key_iv: "42ba2b286d9fbae9161e615955671d41"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.379386 5271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:03.402475 5271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:03.404583 5271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:03.404695 5271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:03.408234 5271 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35275
I20260927 03:54:03.408263 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35275 every 8 connection(s)
I20260927 03:54:03.408510 5271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:03.409404 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:03.409834 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5271
I20260927 03:54:03.409919 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:03.410125 1225 external_mini_cluster.cc:1428] Setting key fe74629d6d690661384948b84557cbec
I20260927 03:54:03.412287 5338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: Bootstrap starting.
I20260927 03:54:03.412828 5338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:03.413036 5338 log.cc:824] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:03.413687 5338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: No bootstrap required, opened a new log
I20260927 03:54:03.415781 5338 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } }
I20260927 03:54:03.415954 5338 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:03.415992 5338 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1defa8a45cd4634af945f1887ada45d, State: Initialized, Role: FOLLOWER
I20260927 03:54:03.416119 5338 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } }
I20260927 03:54:03.416193 5338 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:03.416224 5338 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:03.416270 5338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:03.416797 5338 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } }
I20260927 03:54:03.416904 5338 leader_election.cc:302] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1defa8a45cd4634af945f1887ada45d; no voters:
I20260927 03:54:03.417057 5338 leader_election.cc:288] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:03.417121 5343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:03.417315 5338 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:03.417313 5343 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [term 1 LEADER]: Becoming Leader. State: Replica: d1defa8a45cd4634af945f1887ada45d, State: Running, Role: LEADER
I20260927 03:54:03.417528 5343 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } }
I20260927 03:54:03.417937 5344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1defa8a45cd4634af945f1887ada45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } } }
I20260927 03:54:03.418059 5344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:03.417925 5345 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1defa8a45cd4634af945f1887ada45d. Latest consensus state: current_term: 1 leader_uuid: "d1defa8a45cd4634af945f1887ada45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1defa8a45cd4634af945f1887ada45d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35275 } } }
I20260927 03:54:03.418269 5345 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d [sys.catalog]: This master's current role is: LEADER
W20260927 03:54:03.419081 5358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:03.419171 5358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:03.419418 5352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:03.419601 5352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:03.420702 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.410607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.1.50.126:35275)
0927 03:54:03.410766 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.410775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.410802 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:54:03.411214 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.411221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.411267 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.411382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:03.412130 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.413096 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.414082 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.414483 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.414588 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.417059 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:03.417064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:03.417064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:03.417078 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:03.419022 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.419392 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.419395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.419396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.419497 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.419736 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.419736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.419737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.420029 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:03.420164 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.420396 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.420535 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260927 03:54:03.421202 5352 catalog_manager.cc:1347] Generated new cluster ID: b635ffc83b894d93b568102f0173ad73
I20260927 03:54:03.421249 5352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:03.433487 5352 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:03.434093 5352 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:03.442456 5352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: Generated new TSK 0
I20260927 03:54:03.442636 5352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:54:03.472335 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:03.576131 5364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:03.576354 5364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:03.576377 5364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:03.577621 5364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:03.577665 5364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:03.577692 5364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:03.579272 5364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:03.579510 5364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:03.579746 5364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:03.584666 5369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.584674 5372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.584674 5370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.584831 5364 server_base.cc:1034] running on GCE node
I20260927 03:54:03.585078 5364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:03.585289 5364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:03.586431 5364 hybrid_clock.cc:648] HybridClock initialized: now 1790481243586418 us; error 26 us; skew 500 ppm
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:03.588901 5364 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:03.589210 5364 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:03.589257 5364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:03.589349 5364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:03.590396 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "636abf91aece44f988e7da959d614040"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "91207ffbb1c3c8169b7fa7dfd57a7e4b"
server_key_iv: "2fa5145e73f4d7b763bc2146470248b4"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.590677 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "636abf91aece44f988e7da959d614040"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "91207ffbb1c3c8169b7fa7dfd57a7e4b"
server_key_iv: "2fa5145e73f4d7b763bc2146470248b4"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.591904 5364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:03.592716 5378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.592888 5364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:03.592970 5364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "636abf91aece44f988e7da959d614040"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "91207ffbb1c3c8169b7fa7dfd57a7e4b"
server_key_iv: "2fa5145e73f4d7b763bc2146470248b4"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.593031 5364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:03.607141 5364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:03.609236 5364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:03.609386 5364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:03.609628 5364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:03.609935 5364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:03.609973 5364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.610002 5364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:03.610047 5364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.615368 5364 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:44731
I20260927 03:54:03.615432 5491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:44731 every 8 connection(s)
I20260927 03:54:03.615674 5364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:03.617844 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5364
I20260927 03:54:03.617972 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:03.618219 1225 external_mini_cluster.cc:1428] Setting key bb0a55d19be9e23cb1558df5ff505461
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:54:03.626320 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.616400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:43705 (local address 127.1.50.126:35275)
0927 03:54:03.616475 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.616477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.616610 (+ 133us) server_negotiation.cc:407] Connection header received
0927 03:54:03.617146 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.617147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.617157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.617205 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:03.618437 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.619156 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.620575 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.620780 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.620845 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.623370 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:03.623372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:03.623372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:03.623382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:03.624908 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.625325 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.625327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.625327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.625389 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.625606 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.625607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.625607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.625749 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:03.625818 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.626115 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.626207 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:54:03.626506 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.616335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35275 (local address 127.1.50.65:43705)
0927 03:54:03.616492 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.616531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.617029 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.617273 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.617276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.617530 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:03.618333 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.618340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.619256 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.619257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.620469 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.620472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.620859 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.620859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.620981 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.621558 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:03.621567 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:03.623260 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:03.625014 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.625022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.625024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.625246 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.625454 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.625454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.625455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.625549 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.625818 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:03.625824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:03.625951 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.626086 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.626249 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260927 03:54:03.627082 5492 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35275
I20260927 03:54:03.627183 5492 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:03.627334 5492 heartbeater.cc:511] Master 127.1.50.126:35275 requested a full tablet report, sending...
I20260927 03:54:03.627977 5302 ts_manager.cc:194] Registered new tserver with Master: 636abf91aece44f988e7da959d614040 (127.1.50.65:44731)
I20260927 03:54:03.628697 5302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:43705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:03.645316 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:03.725741 5497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:03.725946 5497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:03.725970 5497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:03.727137 5497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:03.727178 5497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:03.727205 5497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:03.728703 5497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:03.728899 5497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:03.729087 5497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:03.733922 5503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.733927 5505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.733961 5497 server_base.cc:1034] running on GCE node
W20260927 03:54:03.733937 5502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.734357 5497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:03.734604 5497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:03.735770 5497 hybrid_clock.cc:648] HybridClock initialized: now 1790481243735757 us; error 30 us; skew 500 ppm
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:03.738373 5497 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:03.738757 5497 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:03.738811 5497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:03.738930 5497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:03.740005 5497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "5ad1e628d47d4c0494610e1c72c400d0"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "81a3ab75306a6f5862d63d265fc7b462"
server_key_iv: "17eadf25378d206624d37851986bee18"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.740326 5497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "5ad1e628d47d4c0494610e1c72c400d0"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "81a3ab75306a6f5862d63d265fc7b462"
server_key_iv: "17eadf25378d206624d37851986bee18"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.741564 5497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:03.742340 5511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.742496 5497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:03.742580 5497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "5ad1e628d47d4c0494610e1c72c400d0"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "81a3ab75306a6f5862d63d265fc7b462"
server_key_iv: "17eadf25378d206624d37851986bee18"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.742635 5497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:03.754896 5497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:03.756702 5497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:03.756819 5497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:03.757004 5497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:03.757315 5497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:03.757347 5497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.757368 5497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:03.757391 5497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.762703 5497 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:41471
I20260927 03:54:03.762773 5624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:41471 every 8 connection(s)
I20260927 03:54:03.763005 5497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:03.770138 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5497
I20260927 03:54:03.770233 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:03.770431 1225 external_mini_cluster.cc:1428] Setting key ab89815f1a40457248fc170c75ed9e48
I20260927 03:54:03.772169 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.763673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:33349 (local address 127.1.50.126:35275)
0927 03:54:03.763719 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.763721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.763964 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:54:03.764382 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.764382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.764392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.764447 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:03.765311 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.765797 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.767041 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.767231 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.767288 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.769345 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:03.769346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:03.769346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:03.769353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:03.770756 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.771143 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.771143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.771144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.771204 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.771432 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.771432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.771433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.771558 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:03.771671 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.772002 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.772080 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:54:03.772330 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.763646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35275 (local address 127.1.50.66:33349)
0927 03:54:03.763804 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.763844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.764301 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.764504 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.764507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.764762 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:03.765242 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.765247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.765905 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.765906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.766953 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.766956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.767301 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.767302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.767403 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.767925 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:03.767933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:03.769262 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:03.770820 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.770825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.770827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.771089 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.771284 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.771284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.771285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.771381 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.771746 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:03.771752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:03.771874 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.771971 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.772117 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260927 03:54:03.772842 5625 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35275
I20260927 03:54:03.772933 5625 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:03.773077 5625 heartbeater.cc:511] Master 127.1.50.126:35275 requested a full tablet report, sending...
I20260927 03:54:03.773432 5302 ts_manager.cc:194] Registered new tserver with Master: 5ad1e628d47d4c0494610e1c72c400d0 (127.1.50.66:41471)
I20260927 03:54:03.773790 5302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:33349
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:03.796329 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:03.875378 5630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:03.875579 5630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:03.875609 5630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:03.876834 5630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:03.876886 5630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:03.876921 5630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:03.878484 5630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:03.878723 5630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:03.878925 5630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:03.883972 5638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.883975 5635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:03.883972 5636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:03.884217 5630 server_base.cc:1034] running on GCE node
I20260927 03:54:03.884433 5630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:03.884682 5630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:03.885845 5630 hybrid_clock.cc:648] HybridClock initialized: now 1790481243885831 us; error 23 us; skew 500 ppm
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:03.888299 5630 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:03.888693 5630 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:03.888747 5630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:03.888870 5630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:03.889895 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "bd2009664d044122be884997d39f6ae4"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "9b415e4fe8e28c4a58b5ca6699798b39"
server_key_iv: "67a6170f0631e43f60be847f77bd67cf"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.890228 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "bd2009664d044122be884997d39f6ae4"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "9b415e4fe8e28c4a58b5ca6699798b39"
server_key_iv: "67a6170f0631e43f60be847f77bd67cf"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.891489 5630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:03.892344 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.892513 5630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:03.892670 5630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "bd2009664d044122be884997d39f6ae4"
format_stamp: "Formatted at 2026-09-27 03:54:03 on dist-test-slave-96fg"
server_key: "9b415e4fe8e28c4a58b5ca6699798b39"
server_key_iv: "67a6170f0631e43f60be847f77bd67cf"
server_key_version: "encryptionkey@0"
I20260927 03:54:03.892746 5630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:03.912205 5630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:03.914394 5630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:03.914526 5630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:03.914768 5630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:03.915071 5630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:03.915103 5630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.915132 5630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:03.915156 5630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:03.920886 5630 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:34321
I20260927 03:54:03.920943 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:34321 every 8 connection(s)
I20260927 03:54:03.921198 5630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:54:03 dist-test-slave-96fg krb5kdc[5258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:03.930370 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.921885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:35601 (local address 127.1.50.126:35275)
0927 03:54:03.921962 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.921964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.922133 (+ 169us) server_negotiation.cc:407] Connection header received
0927 03:54:03.922634 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.922635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.922663 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.922721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:03.923676 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.924194 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.925248 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.925438 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.925497 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.927654 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:03.927655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:03.927656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:03.927705 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:03.928953 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.929364 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.929365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.929365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.929417 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.929635 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.929636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.929636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.929754 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:03.929818 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.930147 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.930237 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:54:03.930593 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.921829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35275 (local address 127.1.50.67:35601)
0927 03:54:03.921983 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.922046 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.922529 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.922779 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.922783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.923042 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:03.923557 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.923562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.924303 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.924304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.925163 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.925166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.925501 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.925501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.925596 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.926147 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:03.926155 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:03.927558 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:03.929044 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.929051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.929053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.929291 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.929474 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:03.929475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:03.929475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:03.929576 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:03.929811 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:03.929817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:03.929963 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.930095 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.930278 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260927 03:54:03.931249 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5630
I20260927 03:54:03.931336 5758 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35275
I20260927 03:54:03.931403 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:03.931440 5758 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:03.931607 5758 heartbeater.cc:511] Master 127.1.50.126:35275 requested a full tablet report, sending...
I20260927 03:54:03.931941 1225 external_mini_cluster.cc:1428] Setting key b16b7465c2c8a660729fe04cb353a113
I20260927 03:54:03.931974 5302 ts_manager.cc:194] Registered new tserver with Master: bd2009664d044122be884997d39f6ae4 (127.1.50.67:34321)
I20260927 03:54:03.932472 5302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:35601
I20260927 03:54:03.933475 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:54:03.940193 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.934733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.1.50.126:35275)
0927 03:54:03.934813 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.934815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.934911 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:54:03.934963 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.934964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.934974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.935019 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:03.935528 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.936017 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.936744 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.936929 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.936981 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.938104 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:03.938105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:03.938106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:03.938112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:03.939205 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.939494 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.939495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.939495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.939544 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:03.939702 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:03.939703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:03.939703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:03.939838 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:03.939924 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.940024 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.940098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:54:03.941601 5302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:03.943310 5302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:03.949519 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.1.50.66:41471)
0927 03:54:03.945673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.945679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.945689 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:03.945774 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.945776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.945792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.945865 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:03.946245 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947352 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.948633 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949104 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949141 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949190 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.949397 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.949438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260927 03:54:03.949599 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.1.50.65:44731)
0927 03:54:03.946059 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.946063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.946095 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:54:03.946190 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.946193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.946266 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.946334 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:03.946530 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947569 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.948550 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949259 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949335 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949392 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.949467 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.949505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260927 03:54:03.949694 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.1.50.67:34321)
0927 03:54:03.945843 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:03.945847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:03.945954 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:54:03.946035 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:03.946037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:03.946075 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:03.946145 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:03.946278 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947117 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.948625 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949282 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949505 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949562 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:03.949584 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 03:54:03.949621 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260927 03:54:03.949733 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:41471 (local address 127.0.0.1:39516)
0927 03:54:03.945510 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.945557 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.945624 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.946077 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.946080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.946083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:03.946147 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.946148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947445 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.947446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.948546 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.948547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949236 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.949291 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949301 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.949637 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.949678 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":11,"thread_start_us":47,"threads_started":1}
I20260927 03:54:03.950073 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:34321 (local address 127.0.0.1:60752)
0927 03:54:03.945887 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.945896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.945948 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.946157 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.946157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.946167 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:03.946204 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.946205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947210 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.947211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.948544 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.948547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949369 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.949432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949436 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.949464 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.949994 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":493,"thread_start_us":63,"threads_started":1}
I20260927 03:54:03.950165 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:03.945714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:44731 (local address 127.0.0.1:57990)
0927 03:54:03.946049 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:03.946057 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:03.946104 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:03.946412 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:03.946412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:03.946414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:03.946461 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.946463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.947650 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.947651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.948487 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:03.948489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:03.949343 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:03.949343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:03.949396 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:03.949399 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:03.949452 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:54:03.949514 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":49,"threads_started":1}
I20260927 03:54:03.950488 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 (DEFAULT_TABLE table=test-table [id=39b006d56ddf46cfb408a233e8db1ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:03.950830 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c4aca9aa50455cae2bcb174ac2b506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:03.950765 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 (DEFAULT_TABLE table=test-table [id=39b006d56ddf46cfb408a233e8db1ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:03.950811 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 (DEFAULT_TABLE table=test-table [id=39b006d56ddf46cfb408a233e8db1ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:03.951028 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c4aca9aa50455cae2bcb174ac2b506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:03.951079 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c4aca9aa50455cae2bcb174ac2b506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:03.953617 5775 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Bootstrap starting.
I20260927 03:54:03.953944 5776 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Bootstrap starting.
I20260927 03:54:03.954305 5775 tablet_bootstrap.cc:654] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:03.954566 5776 tablet_bootstrap.cc:654] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:03.954900 5775 log.cc:824] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:03.955112 5776 log.cc:824] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:03.955425 5777 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Bootstrap starting.
I20260927 03:54:03.955654 5775 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: No bootstrap required, opened a new log
I20260927 03:54:03.955725 5775 ts_tablet_manager.cc:1397] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:54:03.955798 5776 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: No bootstrap required, opened a new log
I20260927 03:54:03.955861 5776 ts_tablet_manager.cc:1397] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:54:03.955996 5777 tablet_bootstrap.cc:654] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:03.956543 5777 log.cc:824] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:03.957221 5777 tablet_bootstrap.cc:492] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: No bootstrap required, opened a new log
I20260927 03:54:03.957288 5777 ts_tablet_manager.cc:1397] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:03.957423 5776 raft_consensus.cc:348] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.957552 5776 raft_consensus.cc:374] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:03.957582 5776 raft_consensus.cc:729] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636abf91aece44f988e7da959d614040, State: Initialized, Role: FOLLOWER
I20260927 03:54:03.957738 5776 consensus_queue.cc:260] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.957975 5776 ts_tablet_manager.cc:1428] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:54:03.957892 5775 raft_consensus.cc:348] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.958118 5492 heartbeater.cc:503] Master 127.1.50.126:35275 was elected leader, sending a full tablet report...
I20260927 03:54:03.958364 5775 raft_consensus.cc:374] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:03.958427 5775 raft_consensus.cc:729] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2009664d044122be884997d39f6ae4, State: Initialized, Role: FOLLOWER
I20260927 03:54:03.958539 5775 consensus_queue.cc:260] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.958818 5758 heartbeater.cc:503] Master 127.1.50.126:35275 was elected leader, sending a full tablet report...
I20260927 03:54:03.958808 5775 ts_tablet_manager.cc:1428] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:03.959161 5777 raft_consensus.cc:348] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.959307 5777 raft_consensus.cc:374] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:03.959350 5777 raft_consensus.cc:729] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad1e628d47d4c0494610e1c72c400d0, State: Initialized, Role: FOLLOWER
I20260927 03:54:03.959450 5777 consensus_queue.cc:260] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:03.959702 5777 ts_tablet_manager.cc:1428] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:54:03.959708 5625 heartbeater.cc:503] Master 127.1.50.126:35275 was elected leader, sending a full tablet report...
W20260927 03:54:04.013487 5626 tablet.cc:2369] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:54:04.116372 5493 tablet.cc:2369] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:54:04.171751 5759 tablet.cc:2369] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:04.247423 5783 raft_consensus.cc:482] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:04.247552 5783 raft_consensus.cc:504] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:04.247916 5783 leader_election.cc:288] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd2009664d044122be884997d39f6ae4 (127.1.50.67:34321), 636abf91aece44f988e7da959d614040 (127.1.50.65:44731)
I20260927 03:54:04.250774 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.248304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:38049 (local address 127.1.50.65:44731)
0927 03:54:04.248356 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.248358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.248369 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:04.248395 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.248396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.248407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.248455 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:04.248645 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.249086 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250097 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.250451 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250519 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:04.250530 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.250660 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.250697 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:54:04.250809 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.248210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:44731 (local address 127.1.50.66:38049)
0927 03:54:04.248309 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.248317 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.248352 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.248461 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.248462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.248463 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:04.248531 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:04.248532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.249166 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.249166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:04.249994 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:04.249996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.250526 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:04.250593 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:04.250605 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.250649 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.250700 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":28,"thread_start_us":73,"threads_started":1}
I20260927 03:54:04.250828 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.248231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54563 (local address 127.1.50.67:34321)
0927 03:54:04.248284 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.248286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.248302 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:54:04.248331 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.248331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.248343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.248417 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:04.248563 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.249070 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250133 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.250518 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250578 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:04.250588 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.250725 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.250759 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:54:04.251044 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.248111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:34321 (local address 127.1.50.66:54563)
0927 03:54:04.248160 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.248172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.248217 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.248431 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.248432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.248434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:04.248495 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:04.248496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.249173 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.249174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:04.249991 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:04.249994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.250589 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.250590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:04.250667 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:04.250669 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.250713 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.250752 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":28}
I20260927 03:54:04.251133 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c4aca9aa50455cae2bcb174ac2b506" candidate_uuid: "5ad1e628d47d4c0494610e1c72c400d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636abf91aece44f988e7da959d614040" is_pre_election: true
I20260927 03:54:04.251286 5446 raft_consensus.cc:2454] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad1e628d47d4c0494610e1c72c400d0 in term 0.
I20260927 03:54:04.251402 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c4aca9aa50455cae2bcb174ac2b506" candidate_uuid: "5ad1e628d47d4c0494610e1c72c400d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2009664d044122be884997d39f6ae4" is_pre_election: true
I20260927 03:54:04.251502 5514 leader_election.cc:302] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad1e628d47d4c0494610e1c72c400d0, 636abf91aece44f988e7da959d614040; no voters:
I20260927 03:54:04.251539 5712 raft_consensus.cc:2454] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad1e628d47d4c0494610e1c72c400d0 in term 0.
I20260927 03:54:04.251713 5783 raft_consensus.cc:2781] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:04.251780 5783 raft_consensus.cc:482] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:04.251797 5783 raft_consensus.cc:3037] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:04.252486 5783 raft_consensus.cc:504] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:04.252593 5783 leader_election.cc:288] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [CANDIDATE]: Term 1 election: Requested vote from peers bd2009664d044122be884997d39f6ae4 (127.1.50.67:34321), 636abf91aece44f988e7da959d614040 (127.1.50.65:44731)
I20260927 03:54:04.252785 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c4aca9aa50455cae2bcb174ac2b506" candidate_uuid: "5ad1e628d47d4c0494610e1c72c400d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636abf91aece44f988e7da959d614040"
I20260927 03:54:04.252812 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c4aca9aa50455cae2bcb174ac2b506" candidate_uuid: "5ad1e628d47d4c0494610e1c72c400d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2009664d044122be884997d39f6ae4"
I20260927 03:54:04.252856 5446 raft_consensus.cc:3037] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:04.252887 5712 raft_consensus.cc:3037] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:04.253545 5446 raft_consensus.cc:2454] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad1e628d47d4c0494610e1c72c400d0 in term 1.
I20260927 03:54:04.253594 5712 raft_consensus.cc:2454] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad1e628d47d4c0494610e1c72c400d0 in term 1.
I20260927 03:54:04.253727 5514 leader_election.cc:302] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad1e628d47d4c0494610e1c72c400d0, bd2009664d044122be884997d39f6ae4; no voters:
I20260927 03:54:04.253806 5783 raft_consensus.cc:2781] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:04.253955 5783 raft_consensus.cc:686] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad1e628d47d4c0494610e1c72c400d0, State: Running, Role: LEADER
I20260927 03:54:04.254081 5783 consensus_queue.cc:237] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } }
I20260927 03:54:04.254985 5301 catalog_manager.cc:5495] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad1e628d47d4c0494610e1c72c400d0 (127.1.50.66). New cstate: current_term: 1 leader_uuid: "5ad1e628d47d4c0494610e1c72c400d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad1e628d47d4c0494610e1c72c400d0" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:04.302623 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.300168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.1.50.66:41471)
0927 03:54:04.300262 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.300264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.300273 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:04.300305 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.300306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.300325 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.300393 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:04.300527 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.301023 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.301818 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:04.302036 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:04.302091 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:04.302321 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:04.302432 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:04.302509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.302527 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.302546 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 03:54:04.304659 5446 raft_consensus.cc:1261] T 90c4aca9aa50455cae2bcb174ac2b506 P 636abf91aece44f988e7da959d614040 [term 1 FOLLOWER]: Refusing update from remote peer 5ad1e628d47d4c0494610e1c72c400d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:04.304668 5712 raft_consensus.cc:1261] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 1 FOLLOWER]: Refusing update from remote peer 5ad1e628d47d4c0494610e1c72c400d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:04.305016 5783 consensus_queue.cc:1035] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "636abf91aece44f988e7da959d614040" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:04.305184 5785 consensus_queue.cc:1035] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2009664d044122be884997d39f6ae4" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:04.307518 5788 mvcc.cc:205] Tried to move back new op lower bound from 7333811176668581888 to 7333811176465285120. Current Snapshot: MvccSnapshot[applied={T|T < 7333811176668581888}]
I20260927 03:54:04.307711 5790 mvcc.cc:205] Tried to move back new op lower bound from 7333811176668581888 to 7333811176465285120. Current Snapshot: MvccSnapshot[applied={T|T < 7333811176668581888}]
I20260927 03:54:04.307792 5789 mvcc.cc:205] Tried to move back new op lower bound from 7333811176668581888 to 7333811176465285120. Current Snapshot: MvccSnapshot[applied={T|T < 7333811176668581888}]
I20260927 03:54:04.310379 5301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:04.310457 5301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:04.311074 5301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1defa8a45cd4634af945f1887ada45d: Sending DeleteTablet for 3 replicas of tablet 90c4aca9aa50455cae2bcb174ac2b506
I20260927 03:54:04.311371 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:04 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:39516
I20260927 03:54:04.311432 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:04 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:60752
I20260927 03:54:04.311465 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c4aca9aa50455cae2bcb174ac2b506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:04 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:57990
I20260927 03:54:04.311666 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5364
I20260927 03:54:04.316903 5798 tablet_replica.cc:330] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: stopping tablet replica
I20260927 03:54:04.317026 5798 raft_consensus.cc:2229] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:04.317169 5798 raft_consensus.cc:2258] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:04.317353 5799 tablet_replica.cc:330] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: stopping tablet replica
I20260927 03:54:04.317562 5799 raft_consensus.cc:2229] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:04.317726 5799 raft_consensus.cc:2258] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:04.317759 5798 ts_tablet_manager.cc:1905] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:04.318270 5799 ts_tablet_manager.cc:1905] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:04.319742 5798 ts_tablet_manager.cc:1918] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:04.319883 5798 log.cc:1196] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/90c4aca9aa50455cae2bcb174ac2b506
I20260927 03:54:04.319897 5799 ts_tablet_manager.cc:1918] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:04.319986 5799 log.cc:1196] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/90c4aca9aa50455cae2bcb174ac2b506
I20260927 03:54:04.320219 5798 ts_tablet_manager.cc:1939] T 90c4aca9aa50455cae2bcb174ac2b506 P 5ad1e628d47d4c0494610e1c72c400d0: Deleting consensus metadata
I20260927 03:54:04.320343 5799 ts_tablet_manager.cc:1939] T 90c4aca9aa50455cae2bcb174ac2b506 P bd2009664d044122be884997d39f6ae4: Deleting consensus metadata
I20260927 03:54:04.320689 5288 catalog_manager.cc:4841] TS 5ad1e628d47d4c0494610e1c72c400d0 (127.1.50.66:41471): tablet 90c4aca9aa50455cae2bcb174ac2b506 (table test-table [id=39b006d56ddf46cfb408a233e8db1ae1]) successfully deleted
W20260927 03:54:04.320848 5288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:44731 (error 108)
I20260927 03:54:04.320981 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5497
I20260927 03:54:04.321069 5288 catalog_manager.cc:4841] TS bd2009664d044122be884997d39f6ae4 (127.1.50.67:34321): tablet 90c4aca9aa50455cae2bcb174ac2b506 (table test-table [id=39b006d56ddf46cfb408a233e8db1ae1]) successfully deleted
I20260927 03:54:04.322502 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.322377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:44731 (local address 127.0.0.1:58006)
0927 03:54:04.322431 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.322468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:54:04.322633 5288 catalog_manager.cc:4584] TS 636abf91aece44f988e7da959d614040 (127.1.50.65:44731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90c4aca9aa50455cae2bcb174ac2b506: Network error: Client connection negotiation failed: client connection to 127.1.50.65:44731: connect: Connection refused (error 111)
I20260927 03:54:04.325157 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5630
I20260927 03:54:04.329057 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5271
2026-09-27T03:54:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2116 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T03:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:04Z Disabled control of system clock
I20260927 03:54:04.355072 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:41271
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:41271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:54:04.437441 5805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:04.437654 5805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:04.437673 5805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:04.438900 5805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:04.438944 5805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:04.438956 5805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:04.438966 5805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:04.438977 5805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:04.440362 5805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:41271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:04.440585 5805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:04.440852 5805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:04.445401 5813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:04.445413 5811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.445559 5805 server_base.cc:1034] running on GCE node
W20260927 03:54:04.445547 5810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.445925 5805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:04.446228 5805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:04.447388 5805 hybrid_clock.cc:648] HybridClock initialized: now 1790481244447371 us; error 33 us; skew 500 ppm
I20260927 03:54:04.448809 5805 webserver.cc:460] Webserver started at http://127.1.50.126:39779/ using document root <none> and password file <none>
I20260927 03:54:04.449025 5805 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:04.449071 5805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:04.449190 5805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:04.450328 5805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "870b2d2822344025b0891c94b4604e6f"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "5e9996c0e55637e0557c4e23119793e2"
server_key_iv: "100bdc179c85ada1a4fdc09a52d93ef8"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.450652 5805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "870b2d2822344025b0891c94b4604e6f"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "5e9996c0e55637e0557c4e23119793e2"
server_key_iv: "100bdc179c85ada1a4fdc09a52d93ef8"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.451856 5805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:04.452685 5819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.452844 5805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:04.452921 5805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "870b2d2822344025b0891c94b4604e6f"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "5e9996c0e55637e0557c4e23119793e2"
server_key_iv: "100bdc179c85ada1a4fdc09a52d93ef8"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.452991 5805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:04.456439 5805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:04.456710 5805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:04.456820 5805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:04.460484 5805 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:41271
I20260927 03:54:04.460515 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:41271 every 8 connection(s)
I20260927 03:54:04.460835 5805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:04.461509 5872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:04.464282 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: Bootstrap starting.
I20260927 03:54:04.464831 5872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:04.465092 5872 log.cc:824] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:04.465754 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: No bootstrap required, opened a new log
I20260927 03:54:04.467335 5872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } }
I20260927 03:54:04.467473 5872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:04.467494 5872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870b2d2822344025b0891c94b4604e6f, State: Initialized, Role: FOLLOWER
I20260927 03:54:04.467636 5872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } }
I20260927 03:54:04.467697 5872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:04.467718 5872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:04.467746 5872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:04.468228 5872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } }
I20260927 03:54:04.468317 5872 leader_election.cc:302] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 870b2d2822344025b0891c94b4604e6f; no voters:
I20260927 03:54:04.468478 5872 leader_election.cc:288] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:04.468524 5875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:04.468705 5872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:04.468716 5875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [term 1 LEADER]: Becoming Leader. State: Replica: 870b2d2822344025b0891c94b4604e6f, State: Running, Role: LEADER
I20260927 03:54:04.468830 5875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } }
I20260927 03:54:04.469204 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870b2d2822344025b0891c94b4604e6f. Latest consensus state: current_term: 1 leader_uuid: "870b2d2822344025b0891c94b4604e6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } } }
I20260927 03:54:04.469290 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:04.469393 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "870b2d2822344025b0891c94b4604e6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870b2d2822344025b0891c94b4604e6f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 41271 } } }
I20260927 03:54:04.469477 5884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:04.469499 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:04.470070 5884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:04.470602 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5805
I20260927 03:54:04.470717 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:04.470878 1225 external_mini_cluster.cc:1428] Setting key 74b3bceacf7c1dca7f5664093bbdb9c8
I20260927 03:54:04.472990 5884 catalog_manager.cc:1347] Generated new cluster ID: 5df4787861c94ebcafaf5e3227d5127f
I20260927 03:54:04.473042 5884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:04.473337 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.471310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.1.50.126:41271)
0927 03:54:04.471964 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.471972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.472002 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:54:04.472122 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.472129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.472179 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.472316 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.472873 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.472877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.472877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.472881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.472928 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.472943 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.472999 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.473110 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.473168 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":63,"threads_started":1}
I20260927 03:54:04.494673 5884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:04.494858 5884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:04.498167 5884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: Generated new TSK 0
I20260927 03:54:04.498381 5884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:04.504148 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--builtin_ntp_servers=127.1.50.84:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:54:04.626775 5896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:04.627039 5896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:04.627070 5896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:04.628783 5896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:04.628837 5896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:04.628875 5896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:04.630842 5896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:04.631094 5896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:04.631458 5896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:04.636668 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:04.636696 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.636833 5896 server_base.cc:1034] running on GCE node
W20260927 03:54:04.636716 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.637094 5896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:04.637356 5896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:04.638597 5896 hybrid_clock.cc:648] HybridClock initialized: now 1790481244638556 us; error 36 us; skew 500 ppm
I20260927 03:54:04.639828 5896 webserver.cc:460] Webserver started at http://127.1.50.65:36313/ using document root <none> and password file <none>
I20260927 03:54:04.640029 5896 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:04.640070 5896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:04.640168 5896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:04.641153 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "9f3c0f8785ef41aa8426f83325eb97bc"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "49dc2bbe26a78438aacbbce3bea993e3"
server_key_iv: "c5f47b694c2b6d7c2b9345ad2846ba8f"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.641441 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "9f3c0f8785ef41aa8426f83325eb97bc"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "49dc2bbe26a78438aacbbce3bea993e3"
server_key_iv: "c5f47b694c2b6d7c2b9345ad2846ba8f"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.642748 5896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:04.643512 5910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.643716 5896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:04.643810 5896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "9f3c0f8785ef41aa8426f83325eb97bc"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "49dc2bbe26a78438aacbbce3bea993e3"
server_key_iv: "c5f47b694c2b6d7c2b9345ad2846ba8f"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.643872 5896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:04.647234 5896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:04.647490 5896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:04.647607 5896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:04.647805 5896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:04.648087 5896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:04.648132 5896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.648159 5896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:04.648183 5896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.653892 5896 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:35825
I20260927 03:54:04.653932 6023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:35825 every 8 connection(s)
I20260927 03:54:04.654274 5896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:04.657480 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.654960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:58735 (local address 127.1.50.126:41271)
0927 03:54:04.655098 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.655100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.655303 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:54:04.655768 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.655769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.655776 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.655848 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.657122 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.657123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.657124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.657128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.657146 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.657150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.657214 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.657408 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.657425 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:54:04.657598 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.654943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:41271 (local address 127.1.50.65:58735)
0927 03:54:04.655152 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.655189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.655651 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.655916 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.655921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.655930 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.656925 (+ 995us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.656926 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.656959 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.656991 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.657238 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.657245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.657248 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.657343 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.657409 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260927 03:54:04.658226 6024 heartbeater.cc:347] Connected to a master server at 127.1.50.126:41271
I20260927 03:54:04.658324 6024 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:04.658452 6024 heartbeater.cc:511] Master 127.1.50.126:41271 requested a full tablet report, sending...
I20260927 03:54:04.658989 5836 ts_manager.cc:194] Registered new tserver with Master: 9f3c0f8785ef41aa8426f83325eb97bc (127.1.50.65:35825)
I20260927 03:54:04.663450 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 5896
I20260927 03:54:04.663548 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:04.663720 1225 external_mini_cluster.cc:1428] Setting key 63f601940c8dae1280e196c99483b9c9
I20260927 03:54:04.664757 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--builtin_ntp_servers=127.1.50.84:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:54:04.745251 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:04.745474 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:04.745497 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:04.746795 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:04.746835 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:04.746861 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:04.748263 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:04.748481 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:04.748754 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:04.753487 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:04.753487 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.753628 6027 server_base.cc:1034] running on GCE node
W20260927 03:54:04.753484 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.754036 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:04.754274 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:04.755434 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790481244755417 us; error 29 us; skew 500 ppm
I20260927 03:54:04.756680 6027 webserver.cc:460] Webserver started at http://127.1.50.66:43661/ using document root <none> and password file <none>
I20260927 03:54:04.756899 6027 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:04.756947 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:04.757066 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:04.758227 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "d518ac487b66498f993876536f01a5f5"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "4d8fc93cdc7008c3913d0dbff13af271"
server_key_iv: "b58303fdec38821d6f60d41839ec9285"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.758558 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "d518ac487b66498f993876536f01a5f5"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "4d8fc93cdc7008c3913d0dbff13af271"
server_key_iv: "b58303fdec38821d6f60d41839ec9285"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.759882 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:04.760663 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.760879 6027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:04.760964 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "d518ac487b66498f993876536f01a5f5"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "4d8fc93cdc7008c3913d0dbff13af271"
server_key_iv: "b58303fdec38821d6f60d41839ec9285"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.761025 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:04.764361 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:04.764603 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:04.764726 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:04.764910 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:04.765198 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:04.765231 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.765259 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:04.765281 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.771780 6027 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:41361
I20260927 03:54:04.771909 6154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:41361 every 8 connection(s)
I20260927 03:54:04.772195 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:04.775642 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.772917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54091 (local address 127.1.50.126:41271)
0927 03:54:04.772980 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.772983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.773265 (+ 282us) server_negotiation.cc:407] Connection header received
0927 03:54:04.773951 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.773952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.773959 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.774038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.775408 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.775410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.775410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.775415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.775432 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.775436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.775486 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.775554 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.775568 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:54:04.775835 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.772928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:41271 (local address 127.1.50.66:54091)
0927 03:54:04.773125 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.773180 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.773803 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.774112 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.774117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.774128 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.775247 (+ 1119us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.775247 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.775289 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.775327 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.775481 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.775485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.775489 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.775563 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.775626 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260927 03:54:04.776283 6155 heartbeater.cc:347] Connected to a master server at 127.1.50.126:41271
I20260927 03:54:04.776367 6155 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:04.776510 6155 heartbeater.cc:511] Master 127.1.50.126:41271 requested a full tablet report, sending...
I20260927 03:54:04.776844 5836 ts_manager.cc:194] Registered new tserver with Master: d518ac487b66498f993876536f01a5f5 (127.1.50.66:41361)
I20260927 03:54:04.779435 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6027
I20260927 03:54:04.779515 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:04.779657 1225 external_mini_cluster.cc:1428] Setting key 67a5e316f65a22e9bb172795db10d85b
I20260927 03:54:04.780471 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--builtin_ntp_servers=127.1.50.84:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:54:04.859225 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:04.859433 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:04.859454 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:04.860584 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:04.860622 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:04.860647 6158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:04.861969 6158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:04.862219 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:04.862488 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:04.867327 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:04.867329 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:04.867327 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:04.867522 6158 server_base.cc:1034] running on GCE node
I20260927 03:54:04.867794 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:04.868006 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:04.869172 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790481244869157 us; error 32 us; skew 500 ppm
I20260927 03:54:04.870599 6158 webserver.cc:460] Webserver started at http://127.1.50.67:33919/ using document root <none> and password file <none>
I20260927 03:54:04.870815 6158 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:04.870864 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:04.870988 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:04.872048 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "201545428b0742d49f3a317f7072cb72"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "f2d1342a7360bc39ca2c779e55e3aef7"
server_key_iv: "7339c7d62a2bce8d3bfbc7ca3bad729c"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.872387 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "201545428b0742d49f3a317f7072cb72"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "f2d1342a7360bc39ca2c779e55e3aef7"
server_key_iv: "7339c7d62a2bce8d3bfbc7ca3bad729c"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.873724 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:04.874519 6172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.874673 6158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.874758 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "201545428b0742d49f3a317f7072cb72"
format_stamp: "Formatted at 2026-09-27 03:54:04 on dist-test-slave-96fg"
server_key: "f2d1342a7360bc39ca2c779e55e3aef7"
server_key_iv: "7339c7d62a2bce8d3bfbc7ca3bad729c"
server_key_version: "encryptionkey@0"
I20260927 03:54:04.874814 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:04.878118 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:04.878345 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:04.878434 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:04.878583 6158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:04.878829 6158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:04.878858 6158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.878876 6158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:04.878893 6158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:04.884191 6158 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:45907
I20260927 03:54:04.884240 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:45907 every 8 connection(s)
I20260927 03:54:04.884531 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:04.884693 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6158
I20260927 03:54:04.884778 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:04.884974 1225 external_mini_cluster.cc:1428] Setting key d8fb1e00594a9613e0065db47fc984dd
I20260927 03:54:04.887498 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.885260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:41275 (local address 127.1.50.126:41271)
0927 03:54:04.885305 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.885307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.885554 (+ 247us) server_negotiation.cc:407] Connection header received
0927 03:54:04.886146 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.886147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.886153 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.886186 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.887268 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.887269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.887270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.887274 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.887289 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.887293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.887342 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.887414 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.887427 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:04.887679 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.885261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:41271 (local address 127.1.50.67:41275)
0927 03:54:04.885442 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.885488 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.886058 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.886257 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.886261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.886272 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.887119 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.887120 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.887147 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.887176 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.887351 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.887355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.887359 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.887425 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.887486 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 03:54:04.888085 6286 heartbeater.cc:347] Connected to a master server at 127.1.50.126:41271
I20260927 03:54:04.888172 6286 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:04.888307 6286 heartbeater.cc:511] Master 127.1.50.126:41271 requested a full tablet report, sending...
I20260927 03:54:04.888633 5836 ts_manager.cc:194] Registered new tserver with Master: 201545428b0742d49f3a317f7072cb72 (127.1.50.67:45907)
I20260927 03:54:04.896992 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:54:04.899500 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.898439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.1.50.126:41271)
0927 03:54:04.898539 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.898541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.898550 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:04.898614 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.898615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.898620 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.898664 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.899283 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.899284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.899285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.899289 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.899308 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.899312 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.899359 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.899406 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.899422 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 03:54:04.900996 5836 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:04.902725 5836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:04.906787 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.905215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.1.50.65:35825)
0927 03:54:04.905396 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.905402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.905415 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:54:04.905442 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.905444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.905454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.905889 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.906320 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.906326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.906327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.906331 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.906367 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.906387 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.906443 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.906677 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.906694 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260927 03:54:04.906852 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.904997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:35825 (local address 127.0.0.1:51582)
0927 03:54:04.905123 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.905163 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.905254 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.905585 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.905657 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.905666 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.906202 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.906203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.906222 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.906252 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.906453 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.906460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.906463 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.906526 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.906541 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260927 03:54:04.906978 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.905227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.1.50.67:45907)
0927 03:54:04.905453 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.905457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.905632 (+ 175us) server_negotiation.cc:407] Connection header received
0927 03:54:04.905656 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.905658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.905666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.905732 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.906579 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.906586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.906586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.906601 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.906634 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.906659 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.906711 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.906886 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.906901 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260927 03:54:04.906852 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.905003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:45907 (local address 127.0.0.1:57222)
0927 03:54:04.905526 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.905535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.905582 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.905985 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.905985 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.905987 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.906488 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.906488 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.906494 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.906506 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.906735 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.906736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.906737 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.906778 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.906789 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":103,"threads_started":1}
I20260927 03:54:04.907238 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.905221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:41361 (local address 127.0.0.1:51282)
0927 03:54:04.906200 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:04.906209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:04.906256 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:04.906427 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:04.906428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:04.906429 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:04.906777 (+ 348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:04.906777 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:04.906784 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:04.906794 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:04.907118 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:04.907119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:04.907119 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:54:04.907156 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 03:54:04.907169 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":88,"threads_started":1}
I20260927 03:54:04.907423 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:04.905428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.1.50.66:41361)
0927 03:54:04.905602 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:04.905605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:04.906308 (+ 703us) server_negotiation.cc:407] Connection header received
0927 03:54:04.906340 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:04.906342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:04.906356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:04.906418 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:04.906870 (+ 452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:04.906877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:04.906877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:04.906881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:04.907018 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:04.907042 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:04.907096 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:04.907326 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 03:54:04.907343 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260927 03:54:04.907738 6220 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf489e1865c477fad9929f6608f80a7 (DEFAULT_TABLE table=test-table [id=42e25011a5e84d09a5bfe0026b04df87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:04.908054 6220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf489e1865c477fad9929f6608f80a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:04.908071 5958 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf489e1865c477fad9929f6608f80a7 (DEFAULT_TABLE table=test-table [id=42e25011a5e84d09a5bfe0026b04df87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:04.908216 6088 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf489e1865c477fad9929f6608f80a7 (DEFAULT_TABLE table=test-table [id=42e25011a5e84d09a5bfe0026b04df87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:04.908392 5958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf489e1865c477fad9929f6608f80a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:04.908468 6088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf489e1865c477fad9929f6608f80a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:04.910645 6300 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Bootstrap starting.
I20260927 03:54:04.911239 6300 tablet_bootstrap.cc:654] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:04.911466 6300 log.cc:824] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:04.911572 6301 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Bootstrap starting.
I20260927 03:54:04.912132 6301 tablet_bootstrap.cc:654] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:04.912165 6302 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Bootstrap starting.
I20260927 03:54:04.912442 6301 log.cc:824] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:04.912521 6300 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: No bootstrap required, opened a new log
I20260927 03:54:04.912580 6300 ts_tablet_manager.cc:1397] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:04.912781 6302 tablet_bootstrap.cc:654] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:04.913067 6302 log.cc:824] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:04.913188 6301 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: No bootstrap required, opened a new log
I20260927 03:54:04.913265 6301 ts_tablet_manager.cc:1397] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:04.914688 6302 tablet_bootstrap.cc:492] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: No bootstrap required, opened a new log
I20260927 03:54:04.914752 6302 ts_tablet_manager.cc:1397] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:54:04.915391 6300 raft_consensus.cc:348] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.915611 6300 raft_consensus.cc:374] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:04.915645 6300 raft_consensus.cc:729] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201545428b0742d49f3a317f7072cb72, State: Initialized, Role: FOLLOWER
I20260927 03:54:04.915802 6300 consensus_queue.cc:260] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.916091 6300 ts_tablet_manager.cc:1428] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:54:04.916129 6286 heartbeater.cc:503] Master 127.1.50.126:41271 was elected leader, sending a full tablet report...
I20260927 03:54:04.916079 6301 raft_consensus.cc:348] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.916229 6301 raft_consensus.cc:374] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:04.916260 6301 raft_consensus.cc:729] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d518ac487b66498f993876536f01a5f5, State: Initialized, Role: FOLLOWER
I20260927 03:54:04.916499 6301 consensus_queue.cc:260] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.916854 6155 heartbeater.cc:503] Master 127.1.50.126:41271 was elected leader, sending a full tablet report...
I20260927 03:54:04.916839 6301 ts_tablet_manager.cc:1428] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 03:54:04.917064 6302 raft_consensus.cc:348] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.917229 6302 raft_consensus.cc:374] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:04.917273 6302 raft_consensus.cc:729] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3c0f8785ef41aa8426f83325eb97bc, State: Initialized, Role: FOLLOWER
I20260927 03:54:04.917376 6302 consensus_queue.cc:260] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:04.917630 6302 ts_tablet_manager.cc:1428] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:04.917661 6024 heartbeater.cc:503] Master 127.1.50.126:41271 was elected leader, sending a full tablet report...
W20260927 03:54:05.022807 6156 tablet.cc:2369] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:54:05.135764 6287 tablet.cc:2369] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:05.148324 6308 raft_consensus.cc:482] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:05.148459 6308 raft_consensus.cc:504] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:05.148833 6308 leader_election.cc:288] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201545428b0742d49f3a317f7072cb72 (127.1.50.67:45907), d518ac487b66498f993876536f01a5f5 (127.1.50.66:41361)
I20260927 03:54:05.150167 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.149049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:45907 (local address 127.1.50.65:54469)
0927 03:54:05.149100 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.149112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:05.149155 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:05.149371 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:05.149373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:05.149376 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:05.149809 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:05.149809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:05.149816 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:05.149830 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:05.150012 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:05.150013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:05.150035 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 03:54:05.150087 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:54:05.150100 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:54:05.150182 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.149155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:54469 (local address 127.1.50.67:45907)
0927 03:54:05.149223 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.149225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.149236 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:05.149280 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.149281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.149286 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.149362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.149917 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.149919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.149920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.149925 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.149944 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.149948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.150010 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.150095 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.150118 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:54:05.150369 6309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.149160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:41361 (local address 127.1.50.65:52527)
0927 03:54:05.149286 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.149294 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:05.149330 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:05.149440 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:05.149441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:05.149442 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:05.149967 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:05.149968 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:05.149974 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:05.149985 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:05.150172 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:05.150173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:05.150173 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:54:05.150280 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:54:05.150292 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260927 03:54:05.150421 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.149273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:52527 (local address 127.1.50.66:41361)
0927 03:54:05.149341 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.149342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.149351 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:05.149378 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.149379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.149384 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.149431 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.150091 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.150092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.150093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.150097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.150114 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.150118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.150168 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.150344 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.150361 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:54:05.150547 6240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf489e1865c477fad9929f6608f80a7" candidate_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201545428b0742d49f3a317f7072cb72" is_pre_election: true
I20260927 03:54:05.150686 6098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf489e1865c477fad9929f6608f80a7" candidate_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d518ac487b66498f993876536f01a5f5" is_pre_election: true
I20260927 03:54:05.150748 6240 raft_consensus.cc:2454] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3c0f8785ef41aa8426f83325eb97bc in term 0.
I20260927 03:54:05.150825 6098 raft_consensus.cc:2454] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3c0f8785ef41aa8426f83325eb97bc in term 0.
I20260927 03:54:05.150997 5913 leader_election.cc:302] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201545428b0742d49f3a317f7072cb72, 9f3c0f8785ef41aa8426f83325eb97bc; no voters:
I20260927 03:54:05.151154 6308 raft_consensus.cc:2781] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:05.151225 6308 raft_consensus.cc:482] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:05.151249 6308 raft_consensus.cc:3037] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.151928 6308 raft_consensus.cc:504] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
I20260927 03:54:05.152055 6308 leader_election.cc:288] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [CANDIDATE]: Term 1 election: Requested vote from peers 201545428b0742d49f3a317f7072cb72 (127.1.50.67:45907), d518ac487b66498f993876536f01a5f5 (127.1.50.66:41361)
I20260927 03:54:05.152244 6240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf489e1865c477fad9929f6608f80a7" candidate_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201545428b0742d49f3a317f7072cb72"
I20260927 03:54:05.152280 6098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf489e1865c477fad9929f6608f80a7" candidate_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d518ac487b66498f993876536f01a5f5"
I20260927 03:54:05.152344 6098 raft_consensus.cc:3037] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.152344 6240 raft_consensus.cc:3037] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.153324 6240 raft_consensus.cc:2454] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3c0f8785ef41aa8426f83325eb97bc in term 1.
I20260927 03:54:05.153324 6098 raft_consensus.cc:2454] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3c0f8785ef41aa8426f83325eb97bc in term 1.
I20260927 03:54:05.153514 5913 leader_election.cc:302] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3c0f8785ef41aa8426f83325eb97bc, d518ac487b66498f993876536f01a5f5; no voters:
I20260927 03:54:05.153606 6308 raft_consensus.cc:2781] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:05.153759 6308 raft_consensus.cc:686] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 1 LEADER]: Becoming Leader. State: Replica: 9f3c0f8785ef41aa8426f83325eb97bc, State: Running, Role: LEADER
I20260927 03:54:05.153846 6308 consensus_queue.cc:237] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } }
W20260927 03:54:05.154875 6025 tablet.cc:2369] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:05.154815 5836 catalog_manager.cc:5495] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f3c0f8785ef41aa8426f83325eb97bc (127.1.50.65). New cstate: current_term: 1 leader_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f3c0f8785ef41aa8426f83325eb97bc" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:05.193472 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.192688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.1.50.65:35825)
0927 03:54:05.192741 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.192742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.192750 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:05.192820 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.192821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.192834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.192876 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.193300 (+ 424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.193301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.193301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.193304 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.193318 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.193322 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.193360 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.193389 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.193399 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:54:05.195263 6240 raft_consensus.cc:1261] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 1 FOLLOWER]: Refusing update from remote peer 9f3c0f8785ef41aa8426f83325eb97bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:05.195268 6098 raft_consensus.cc:1261] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 1 FOLLOWER]: Refusing update from remote peer 9f3c0f8785ef41aa8426f83325eb97bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:05.195608 6310 consensus_queue.cc:1035] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201545428b0742d49f3a317f7072cb72" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:05.195734 6308 consensus_queue.cc:1035] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d518ac487b66498f993876536f01a5f5" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:05.197927 6313 mvcc.cc:205] Tried to move back new op lower bound from 7333811180317077504 to 7333811180150824960. Current Snapshot: MvccSnapshot[applied={T|T < 7333811180317077504}]
I20260927 03:54:05.200419 5836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:05.200518 5836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:05.201222 5836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 870b2d2822344025b0891c94b4604e6f: Sending DeleteTablet for 3 replicas of tablet 1bf489e1865c477fad9929f6608f80a7
I20260927 03:54:05.201568 6088 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf489e1865c477fad9929f6608f80a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:05 UTC) from {username='slave'} at 127.0.0.1:51282
I20260927 03:54:05.201576 6220 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf489e1865c477fad9929f6608f80a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:05 UTC) from {username='slave'} at 127.0.0.1:57222
I20260927 03:54:05.201659 5958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf489e1865c477fad9929f6608f80a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:05 UTC) from {username='slave'} at 127.0.0.1:51582
I20260927 03:54:05.201862 6324 tablet_replica.cc:330] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: stopping tablet replica
I20260927 03:54:05.201956 6325 tablet_replica.cc:330] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc: stopping tablet replica
I20260927 03:54:05.202066 6324 raft_consensus.cc:2229] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:05.202132 6325 raft_consensus.cc:2229] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:05.202224 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5896
I20260927 03:54:05.202262 6325 raft_consensus.cc:2258] T 1bf489e1865c477fad9929f6608f80a7 P 9f3c0f8785ef41aa8426f83325eb97bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:05.202343 6324 raft_consensus.cc:2258] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:05.203112 6324 ts_tablet_manager.cc:1905] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:05.204857 6324 ts_tablet_manager.cc:1918] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:05.204931 6324 log.cc:1196] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/1bf489e1865c477fad9929f6608f80a7
I20260927 03:54:05.205163 6324 ts_tablet_manager.cc:1939] T 1bf489e1865c477fad9929f6608f80a7 P 201545428b0742d49f3a317f7072cb72: Deleting consensus metadata
I20260927 03:54:05.205798 5822 catalog_manager.cc:4841] TS 201545428b0742d49f3a317f7072cb72 (127.1.50.67:45907): tablet 1bf489e1865c477fad9929f6608f80a7 (table test-table [id=42e25011a5e84d09a5bfe0026b04df87]) successfully deleted
I20260927 03:54:05.206039 6323 tablet_replica.cc:330] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: stopping tablet replica
I20260927 03:54:05.206166 6323 raft_consensus.cc:2229] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:05.206269 6323 raft_consensus.cc:2258] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:05.206676 6323 ts_tablet_manager.cc:1905] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:05.208099 6323 ts_tablet_manager.cc:1918] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:05.208176 6323 log.cc:1196] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/1bf489e1865c477fad9929f6608f80a7
I20260927 03:54:05.208400 6323 ts_tablet_manager.cc:1939] T 1bf489e1865c477fad9929f6608f80a7 P d518ac487b66498f993876536f01a5f5: Deleting consensus metadata
I20260927 03:54:05.208853 5822 catalog_manager.cc:4841] TS d518ac487b66498f993876536f01a5f5 (127.1.50.66:41361): tablet 1bf489e1865c477fad9929f6608f80a7 (table test-table [id=42e25011a5e84d09a5bfe0026b04df87]) successfully deleted
W20260927 03:54:05.209016 5820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:35825 (error 108)
I20260927 03:54:05.209470 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.209348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:35825 (local address 127.0.0.1:51606)
0927 03:54:05.209396 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.209436 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:35825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:54:05.209498 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6027
W20260927 03:54:05.209599 5820 catalog_manager.cc:4584] TS 9f3c0f8785ef41aa8426f83325eb97bc (127.1.50.65:35825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bf489e1865c477fad9929f6608f80a7: Network error: Client connection negotiation failed: client connection to 127.1.50.65:35825: connect: Connection refused (error 111)
I20260927 03:54:05.213605 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6158
I20260927 03:54:05.217800 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 5805
2026-09-27T03:54:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (888 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 03:54:05.236194 1225 mini_oidc.cc:152] Starting JWKS server
I20260927 03:54:05.237219 1225 webserver.cc:460] Webserver started at https://127.0.0.1:35661/ using document root <none> and password file <none>
I20260927 03:54:05.237298 1225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:54:05.237450 1225 webserver.cc:460] Webserver started at http://127.0.0.1:34531/ using document root <none> and password file <none>
2026-09-27T03:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:05Z Disabled control of system clock
I20260927 03:54:05.244730 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35865
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35661/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/testchainca.cert with env {}
W20260927 03:54:05.323334 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:05.323532 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:05.323553 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:05.324654 6332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:05.324689 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:05.324700 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:05.324712 6332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:05.324723 6332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:54:05.324733 6332 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:54:05.324743 6332 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35661/jwks.json
I20260927 03:54:05.326218 6332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35661/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:05.326411 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:05.326571 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:05.331400 6340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.331421 6338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.331458 6337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.331463 6332 server_base.cc:1034] running on GCE node
I20260927 03:54:05.331964 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:05.332257 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:05.333403 6332 hybrid_clock.cc:648] HybridClock initialized: now 1790481245333386 us; error 35 us; skew 500 ppm
I20260927 03:54:05.334790 6332 webserver.cc:460] Webserver started at http://127.1.50.126:40449/ using document root <none> and password file <none>
I20260927 03:54:05.335033 6332 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:05.335084 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:05.335204 6332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:05.336243 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "d63bab55a0e548e0bb9d4870837f0304"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "a0a7860c280ab98fa1df700d8e8a373f"
server_key_iv: "ca4141475553d386b921b78969a565ab"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.336570 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "d63bab55a0e548e0bb9d4870837f0304"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "a0a7860c280ab98fa1df700d8e8a373f"
server_key_iv: "ca4141475553d386b921b78969a565ab"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.337790 6332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:05.338564 6346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:05.338791 6332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:05.338876 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "d63bab55a0e548e0bb9d4870837f0304"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "a0a7860c280ab98fa1df700d8e8a373f"
server_key_iv: "ca4141475553d386b921b78969a565ab"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.338943 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:05.364228 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:05.364552 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:05.364683 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:05.368387 6332 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35865
I20260927 03:54:05.368418 6398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35865 every 8 connection(s)
I20260927 03:54:05.368743 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:05.369056 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6332
I20260927 03:54:05.369158 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:05.369352 1225 external_mini_cluster.cc:1428] Setting key 8a8dac26022093a58bf55a27a4a01d15
I20260927 03:54:05.369678 6399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:05.372438 6399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: Bootstrap starting.
I20260927 03:54:05.373014 6399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:05.373270 6399 log.cc:824] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:05.373864 6399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: No bootstrap required, opened a new log
I20260927 03:54:05.374694 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.369805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.1.50.126:35865)
0927 03:54:05.370052 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.370061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.370095 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:54:05.370767 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.370775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.370839 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.370962 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.371171 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.372041 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.372851 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.373289 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.373400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.374012 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.374034 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.374035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.374209 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.374241 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.374253 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.374299 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.374422 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.374482 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260927 03:54:05.375643 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:54:05.375931 6399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } }
I20260927 03:54:05.376097 6399 raft_consensus.cc:374] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:05.376129 6399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63bab55a0e548e0bb9d4870837f0304, State: Initialized, Role: FOLLOWER
I20260927 03:54:05.376250 6399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } }
I20260927 03:54:05.376324 6399 raft_consensus.cc:388] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:05.376354 6399 raft_consensus.cc:482] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:05.376394 6399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.376974 6399 raft_consensus.cc:504] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } }
I20260927 03:54:05.377064 6399 leader_election.cc:302] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d63bab55a0e548e0bb9d4870837f0304; no voters:
I20260927 03:54:05.377229 6399 leader_election.cc:288] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:05.377274 6404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:05.377409 6404 raft_consensus.cc:686] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [term 1 LEADER]: Becoming Leader. State: Replica: d63bab55a0e548e0bb9d4870837f0304, State: Running, Role: LEADER
I20260927 03:54:05.377465 6399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:05.377491 6404 consensus_queue.cc:237] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } }
I20260927 03:54:05.377840 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d63bab55a0e548e0bb9d4870837f0304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } } }
I20260927 03:54:05.377883 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d63bab55a0e548e0bb9d4870837f0304. Latest consensus state: current_term: 1 leader_uuid: "d63bab55a0e548e0bb9d4870837f0304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63bab55a0e548e0bb9d4870837f0304" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35865 } } }
I20260927 03:54:05.377928 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:05.377954 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304 [sys.catalog]: This master's current role is: LEADER
W20260927 03:54:05.379022 6420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:05.379074 6420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:05.379124 6411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:05.379245 6411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:05.380606 6411 catalog_manager.cc:1347] Generated new cluster ID: 38342191f0004af0bd499595e115a79c
I20260927 03:54:05.380653 6411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:05.391959 6411 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:05.392669 6411 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:05.399683 6411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d63bab55a0e548e0bb9d4870837f0304: Generated new TSK 0
I20260927 03:54:05.399885 6411 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:54:05.429388 6429 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 03:54:05.429513 6429 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:54:05.429766 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.413855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.1.50.126:35865)
0927 03:54:05.413907 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.413909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.414050 (+ 141us) server_negotiation.cc:407] Connection header received
0927 03:54:05.414161 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.414162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.414172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.414229 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:54:05.414411 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.414957 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.416121 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.416316 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.416378 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.416644 (+ 266us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:54:05.429504 (+ 12860us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:54:05.429694 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37942: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 03:54:05.429826 6402 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37942: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 03:54:05.430192 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:05.413778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35865 (local address 127.0.0.1:37942)
0927 03:54:05.413959 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.413970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:05.414068 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:05.414251 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:05.414252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:05.414255 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:54:05.414324 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.414327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.415032 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.415033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.416025 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.416028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.416393 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.416394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.416458 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.416484 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:54:05.429723 (+ 13239us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:54:05.429846 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.50.126:35865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.50.126:35865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 03:54:05.430729 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6332
2026-09-27T03:54:05Z chronyd exiting
I20260927 03:54:05.466464 1225 test_util.cc:175] -----------------------------------------------
I20260927 03:54:05.466563 1225 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481222547165-1225-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 03:54:05.467356 1225 mini_oidc.cc:152] Starting JWKS server
I20260927 03:54:05.468030 1225 webserver.cc:460] Webserver started at https://127.0.0.1:42671/ using document root <none> and password file <none>
I20260927 03:54:05.468077 1225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:54:05.468196 1225 webserver.cc:460] Webserver started at http://127.0.0.1:46211/ using document root <none> and password file <none>
2026-09-27T03:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:05Z Disabled control of system clock
I20260927 03:54:05.475347 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34357
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34357
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/testchainca.cert with env {}
W20260927 03:54:05.554975 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:05.555210 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:05.555233 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:05.556478 6435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:05.556522 6435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:05.556533 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:05.556545 6435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:05.556555 6435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:54:05.556566 6435 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:54:05.556578 6435 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42671/jwks.json
I20260927 03:54:05.558178 6435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:05.558409 6435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:05.558614 6435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:05.563167 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.563211 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.563211 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.563367 6435 server_base.cc:1034] running on GCE node
I20260927 03:54:05.563593 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:05.563872 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:05.565002 6435 hybrid_clock.cc:648] HybridClock initialized: now 1790481245564981 us; error 40 us; skew 500 ppm
I20260927 03:54:05.566164 6435 webserver.cc:460] Webserver started at http://127.1.50.126:45995/ using document root <none> and password file <none>
I20260927 03:54:05.566349 6435 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:05.566388 6435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:05.566478 6435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:05.567477 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "d9f20ae993104c2c9a58e17e1b6475f2"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "c9ebe456fea262b4b9e449eddec75e3b"
server_key_iv: "79a0bec2ba7524dc5efff643c945819d"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.567847 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "d9f20ae993104c2c9a58e17e1b6475f2"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "c9ebe456fea262b4b9e449eddec75e3b"
server_key_iv: "79a0bec2ba7524dc5efff643c945819d"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.569043 6435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:05.569778 6449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:05.569958 6435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:05.570070 6435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "d9f20ae993104c2c9a58e17e1b6475f2"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "c9ebe456fea262b4b9e449eddec75e3b"
server_key_iv: "79a0bec2ba7524dc5efff643c945819d"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.570132 6435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:05.579850 6435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:05.580144 6435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:05.580245 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:05.584066 6435 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34357
I20260927 03:54:05.584100 6501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34357 every 8 connection(s)
I20260927 03:54:05.584411 6435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:05.585368 6502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:05.587653 6502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: Bootstrap starting.
I20260927 03:54:05.588181 6502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:05.588410 6502 log.cc:824] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:05.589039 6502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: No bootstrap required, opened a new log
I20260927 03:54:05.589226 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6435
I20260927 03:54:05.589314 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:05.589463 1225 external_mini_cluster.cc:1428] Setting key e3c1ce7cd488489e93ce63c7f4ed7411
I20260927 03:54:05.590924 6502 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } }
I20260927 03:54:05.591066 6502 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:05.591099 6502 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f20ae993104c2c9a58e17e1b6475f2, State: Initialized, Role: FOLLOWER
I20260927 03:54:05.591257 6502 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } }
I20260927 03:54:05.591341 6502 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:05.591365 6502 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:05.591393 6502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.591884 6502 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } }
I20260927 03:54:05.591966 6502 leader_election.cc:302] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f20ae993104c2c9a58e17e1b6475f2; no voters:
I20260927 03:54:05.592104 6502 leader_election.cc:288] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:05.592176 6507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:05.592305 6507 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [term 1 LEADER]: Becoming Leader. State: Replica: d9f20ae993104c2c9a58e17e1b6475f2, State: Running, Role: LEADER
I20260927 03:54:05.592373 6502 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:05.592396 6507 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } }
I20260927 03:54:05.592713 6507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f20ae993104c2c9a58e17e1b6475f2. Latest consensus state: current_term: 1 leader_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } } }
I20260927 03:54:05.592768 6507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:05.592730 6508 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20ae993104c2c9a58e17e1b6475f2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34357 } } }
I20260927 03:54:05.592800 6508 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2 [sys.catalog]: This master's current role is: LEADER
W20260927 03:54:05.594045 6523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:05.594098 6523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:05.594136 6513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:05.594266 6513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:05.596068 6513 catalog_manager.cc:1347] Generated new cluster ID: c9b1f4cdfae1479698d4c22f75cd5ba8
I20260927 03:54:05.596092 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.589967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.1.50.126:34357)
0927 03:54:05.590150 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.590158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.590183 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:54:05.590892 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.590898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.590954 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.591061 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.591300 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.592824 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.593761 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.594399 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.594513 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.595393 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.595397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.595397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.595588 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.595629 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.595644 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.595725 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.595801 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.595883 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260927 03:54:05.596122 6513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:05.598786 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:54:05.603147 6513 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:05.603881 6513 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:05.613780 6513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f20ae993104c2c9a58e17e1b6475f2: Generated new TSK 0
I20260927 03:54:05.613999 6513 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:54:05.645661 6532 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 03:54:05.645757 6532 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:54:05.646138 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.623680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.1.50.126:34357)
0927 03:54:05.623725 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.623727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.623791 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:54:05.623886 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.623887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.623898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.623960 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:54:05.624105 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.624589 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.625794 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.626006 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.626104 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.626291 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:54:05.645888 (+ 19597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:54:05.646082 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:54:05.646188 6506 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 03:54:05.646270 6531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:05.623582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34357 (local address 127.0.0.1:43800)
0927 03:54:05.623719 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.623733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:05.623811 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:05.623976 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:05.623977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:05.623979 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:54:05.624041 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.624044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.624665 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.624666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.625640 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.625643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.626113 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.626114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.626185 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.626192 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:54:05.646039 (+ 19847us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:54:05.646158 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.50.126:34357: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260927 03:54:05.646624 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6435
2026-09-27T03:54:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 03:54:05.683816 1225 mini_oidc.cc:152] Starting JWKS server
I20260927 03:54:05.684517 1225 webserver.cc:460] Webserver started at https://127.0.0.1:35887/ using document root <none> and password file <none>
I20260927 03:54:05.684562 1225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:54:05.684700 1225 webserver.cc:460] Webserver started at http://127.0.0.1:40229/ using document root <none> and password file <none>
2026-09-27T03:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:05Z Disabled control of system clock
I20260927 03:54:05.691993 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43305
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:43305
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35887/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:54:05.771618 6538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:05.771844 6538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:05.771875 6538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:05.773084 6538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:05.773130 6538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:05.773146 6538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:05.773164 6538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:05.773177 6538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:54:05.773195 6538 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:54:05.773213 6538 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35887/jwks.json
I20260927 03:54:05.774662 6538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:43305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35887/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:05.774927 6538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:05.775205 6538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:05.780282 6543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.780270 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.780303 6538 server_base.cc:1034] running on GCE node
W20260927 03:54:05.780274 6546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.780910 6538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:05.781221 6538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:05.782390 6538 hybrid_clock.cc:648] HybridClock initialized: now 1790481245782377 us; error 32 us; skew 500 ppm
I20260927 03:54:05.783708 6538 webserver.cc:460] Webserver started at http://127.1.50.126:34293/ using document root <none> and password file <none>
I20260927 03:54:05.783987 6538 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:05.784039 6538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:05.784169 6538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:05.785405 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "ae13b331da4648b7b2070705e0045551"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "3a0d5f5ef6ee790f8bce2cf60b3fb292"
server_key_iv: "6bfab63a7a6f41327fa46512ed4e7473"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.785794 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "ae13b331da4648b7b2070705e0045551"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "3a0d5f5ef6ee790f8bce2cf60b3fb292"
server_key_iv: "6bfab63a7a6f41327fa46512ed4e7473"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.787114 6538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:05.788023 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:05.788211 6538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:05.788319 6538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "ae13b331da4648b7b2070705e0045551"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "3a0d5f5ef6ee790f8bce2cf60b3fb292"
server_key_iv: "6bfab63a7a6f41327fa46512ed4e7473"
server_key_version: "encryptionkey@0"
I20260927 03:54:05.788408 6538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:05.803131 6538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:05.803469 6538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:05.803617 6538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:05.807742 6538 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:43305
I20260927 03:54:05.807801 6604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:43305 every 8 connection(s)
I20260927 03:54:05.808135 6538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:05.808795 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:05.811391 6605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: Bootstrap starting.
I20260927 03:54:05.811928 6605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:05.812186 6605 log.cc:824] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:05.812796 6605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: No bootstrap required, opened a new log
I20260927 03:54:05.814400 6605 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } }
I20260927 03:54:05.814534 6605 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:05.814555 6605 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae13b331da4648b7b2070705e0045551, State: Initialized, Role: FOLLOWER
I20260927 03:54:05.814702 6605 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } }
I20260927 03:54:05.814764 6605 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:05.814785 6605 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:05.814828 6605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:05.815331 6605 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } }
I20260927 03:54:05.815418 6605 leader_election.cc:302] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae13b331da4648b7b2070705e0045551; no voters:
I20260927 03:54:05.815569 6605 leader_election.cc:288] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:05.815650 6608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:05.815809 6605 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:05.815812 6608 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [term 1 LEADER]: Becoming Leader. State: Replica: ae13b331da4648b7b2070705e0045551, State: Running, Role: LEADER
I20260927 03:54:05.815955 6608 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } }
I20260927 03:54:05.816220 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6538
I20260927 03:54:05.816314 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:05.816483 1225 external_mini_cluster.cc:1428] Setting key 10277574dcc45325a1e406dc211598b8
I20260927 03:54:05.816999 6610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae13b331da4648b7b2070705e0045551. Latest consensus state: current_term: 1 leader_uuid: "ae13b331da4648b7b2070705e0045551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } } }
I20260927 03:54:05.817085 6610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:05.817277 6609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae13b331da4648b7b2070705e0045551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13b331da4648b7b2070705e0045551" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43305 } } }
I20260927 03:54:05.817499 6609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551 [sys.catalog]: This master's current role is: LEADER
W20260927 03:54:05.817636 6626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:05.817698 6626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:05.817756 6625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:05.817909 6625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:05.819409 6625 catalog_manager.cc:1347] Generated new cluster ID: 2e400b6c48dd49b889d3861d5504eba7
I20260927 03:54:05.819460 6625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:05.821982 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.816960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.1.50.126:43305)
0927 03:54:05.817452 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.817461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.817643 (+ 182us) server_negotiation.cc:407] Connection header received
0927 03:54:05.817823 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.817830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.817896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.818011 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:05.818218 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.819355 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.820132 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.820555 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.820696 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.821323 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:05.821326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:05.821327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:05.821491 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:05.821526 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:05.821536 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:05.821591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:05.821682 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:54:05.821763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":55,"threads_started":1}
I20260927 03:54:05.826985 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:54:05.839178 6625 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:05.839834 6625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:05.847605 6625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae13b331da4648b7b2070705e0045551: Generated new TSK 0
I20260927 03:54:05.847882 6625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:05.860447 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:05.856601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.1.50.126:43305)
0927 03:54:05.856647 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:05.856650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:05.856771 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:54:05.856819 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:05.856820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:05.856831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:05.856899 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:54:05.857065 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.857584 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.858829 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.859023 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.859087 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.859300 (+ 213us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:54:05.860221 (+ 921us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35887/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:54:05.860352 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51704: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35887/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 03:54:05.860505 6634 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:05.856510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43305 (local address 127.0.0.1:51704)
0927 03:54:05.856654 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:05.856667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:05.856753 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:05.856912 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:05.856913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:05.856915 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:54:05.856972 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.856974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.857694 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.857695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.858722 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:05.858725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:05.859101 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:05.859102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:05.859170 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:05.859176 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:54:05.860318 (+ 1142us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35887/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:54:05.860411 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.50.126:43305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35887/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
W20260927 03:54:05.860522 6622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51704: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35887/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 03:54:05.860924 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6538
E20260927 03:54:05.871927 6635 webserver.cc:533] Webserver: sslize failed:
E20260927 03:54:05.872022 6635 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:54:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T03:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:05Z Disabled control of system clock
I20260927 03:54:05.905483 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35767
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35767
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/testchainca.cert with env {}
W20260927 03:54:05.989018 6639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:05.989248 6639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:05.989274 6639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:05.990516 6639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:05.990569 6639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:05.990583 6639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:05.990597 6639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:05.990610 6639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:54:05.990623 6639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:54:05.990641 6639 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 03:54:05.992154 6639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:05.992376 6639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:05.992558 6639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:05.997448 6647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.997474 6639 server_base.cc:1034] running on GCE node
W20260927 03:54:05.997633 6645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:05.997546 6644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:05.997903 6639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:05.998211 6639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:05.999394 6639 hybrid_clock.cc:648] HybridClock initialized: now 1790481245999378 us; error 38 us; skew 500 ppm
I20260927 03:54:06.000640 6639 webserver.cc:460] Webserver started at http://127.1.50.126:37715/ using document root <none> and password file <none>
I20260927 03:54:06.000914 6639 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:06.000967 6639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:06.001094 6639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:06.002214 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "453b6104afd443199e8365fc80044fa7"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "8cdb06741b602a99d4bb8797a605751a"
server_key_iv: "6c5221ce88f44e5de8c91baa26e53734"
server_key_version: "encryptionkey@0"
I20260927 03:54:06.002547 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "453b6104afd443199e8365fc80044fa7"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "8cdb06741b602a99d4bb8797a605751a"
server_key_iv: "6c5221ce88f44e5de8c91baa26e53734"
server_key_version: "encryptionkey@0"
I20260927 03:54:06.003880 6639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:06.005152 6653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:06.005347 6639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:06.005443 6639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "453b6104afd443199e8365fc80044fa7"
format_stamp: "Formatted at 2026-09-27 03:54:05 on dist-test-slave-96fg"
server_key: "8cdb06741b602a99d4bb8797a605751a"
server_key_iv: "6c5221ce88f44e5de8c91baa26e53734"
server_key_version: "encryptionkey@0"
I20260927 03:54:06.005518 6639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:06.024976 6639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:06.025327 6639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:06.025458 6639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:06.029435 6639 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35767
I20260927 03:54:06.029470 6705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35767 every 8 connection(s)
I20260927 03:54:06.029806 6639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:06.030088 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6639
I20260927 03:54:06.030186 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:06.030323 1225 external_mini_cluster.cc:1428] Setting key a6f12c5e314a00b3fe91adbd8c2f5f30
I20260927 03:54:06.030884 6706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:06.033869 6706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7: Bootstrap starting.
I20260927 03:54:06.034488 6706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:06.034749 6706 log.cc:824] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:06.035362 6706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7: No bootstrap required, opened a new log
I20260927 03:54:06.035971 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:06.030789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.1.50.126:35767)
0927 03:54:06.030979 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:06.030989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:06.031017 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:54:06.031446 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:06.031453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:06.031513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:06.031630 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:06.031849 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.033096 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.034144 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.034581 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.034756 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:06.035353 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:06.035357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:06.035358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:06.035538 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:06.035571 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:06.035582 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:06.035642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:06.035712 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:54:06.035791 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 03:54:06.036793 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:54:06.037310 6706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:54:06.037467 6706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:06.037503 6706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453b6104afd443199e8365fc80044fa7, State: Initialized, Role: FOLLOWER
I20260927 03:54:06.037629 6706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:54:06.037700 6706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:06.037730 6706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:06.037777 6706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:06.038344 6706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:54:06.038446 6706 leader_election.cc:302] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453b6104afd443199e8365fc80044fa7; no voters:
I20260927 03:54:06.038589 6706 leader_election.cc:288] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:06.038650 6711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:06.038784 6711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 453b6104afd443199e8365fc80044fa7, State: Running, Role: LEADER
I20260927 03:54:06.038830 6706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:06.038872 6711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:54:06.039222 6713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453b6104afd443199e8365fc80044fa7. Latest consensus state: current_term: 1 leader_uuid: "453b6104afd443199e8365fc80044fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } } }
I20260927 03:54:06.039230 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453b6104afd443199e8365fc80044fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453b6104afd443199e8365fc80044fa7" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } } }
I20260927 03:54:06.039387 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:06.039355 6713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:06.039731 6720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:06.040205 6720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:06.041553 6720 catalog_manager.cc:1347] Generated new cluster ID: a3a6ad46b4684a36889b076142ae2a22
I20260927 03:54:06.041601 6720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:06.070359 6720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:06.071120 6720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:06.076915 6720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 453b6104afd443199e8365fc80044fa7: Generated new TSK 0
I20260927 03:54:06.077129 6720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:06.117336 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:06.081766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.1.50.126:35767)
0927 03:54:06.081806 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:06.081808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:06.081862 (+ 54us) server_negotiation.cc:407] Connection header received
0927 03:54:06.081947 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:06.081948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:06.081960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:06.082047 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:54:06.082186 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.082678 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.083805 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.084008 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.084052 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:06.084228 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:54:06.117099 (+ 32871us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:54:06.117269 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:54:06.117404 6709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 03:54:06.117563 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:06.081669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35767 (local address 127.0.0.1:52428)
0927 03:54:06.081800 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:06.081811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:06.081881 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:06.082052 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:06.082053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:06.082055 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:54:06.082126 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:06.082129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.082764 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.082766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:06.083722 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:06.083724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:06.084072 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:06.084073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:06.084139 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:06.084144 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:54:06.117279 (+ 33135us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:54:06.117434 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.50.126:35767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260927 03:54:06.117868 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6639
2026-09-27T03:54:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (236 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:06 dist-test-slave-96fg krb5kdc[6747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:06 dist-test-slave-96fg krb5kdc[6747](info): set up 2 sockets
Sep 27 03:54:06 dist-test-slave-96fg krb5kdc[6747](info): commencing operation
krb5kdc: starting...
W20260927 03:54:07.258380 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:07.372761 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:33587
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:07.451346 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:07.451547 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:07.451574 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:07.452704 6760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:07.452749 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:07.452768 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:07.452785 6760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:07.452801 6760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:07.454222 6760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:07.454444 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:07.454633 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:07.459285 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:07.459390 6760 server_base.cc:1034] running on GCE node
W20260927 03:54:07.459319 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.459280 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:07.459738 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:07.460011 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:07.461174 6760 hybrid_clock.cc:648] HybridClock initialized: now 1790481247461152 us; error 35 us; skew 500 ppm
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:07.466472 6760 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:07.466872 6760 webserver.cc:460] Webserver started at http://127.1.50.126:42345/ using document root <none> and password file <none>
I20260927 03:54:07.467041 6760 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:07.467078 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:07.467166 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:07.468110 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "a3a6f214c9d941d3a383f44ac28e8aaf"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "81f3af8003a7ab9a6632dbe22d06537a"
server_key_iv: "01de115748fa7d9a36aaec4e5f603051"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.468393 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "a3a6f214c9d941d3a383f44ac28e8aaf"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "81f3af8003a7ab9a6632dbe22d06537a"
server_key_iv: "01de115748fa7d9a36aaec4e5f603051"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.469478 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:07.470116 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.470263 6760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:07.470336 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "a3a6f214c9d941d3a383f44ac28e8aaf"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "81f3af8003a7ab9a6632dbe22d06537a"
server_key_iv: "01de115748fa7d9a36aaec4e5f603051"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.470405 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:07.504755 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:07.506804 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:07.506969 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:07.510710 6760 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:33587
I20260927 03:54:07.510736 6827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:33587 every 8 connection(s)
I20260927 03:54:07.511065 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:07.511979 6828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:07.514245 6828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf: Bootstrap starting.
I20260927 03:54:07.514783 6828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:07.515031 6828 log.cc:824] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:07.515635 6828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf: No bootstrap required, opened a new log
I20260927 03:54:07.517162 6828 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } }
I20260927 03:54:07.517311 6828 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:07.517334 6828 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a6f214c9d941d3a383f44ac28e8aaf, State: Initialized, Role: FOLLOWER
I20260927 03:54:07.517463 6828 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } }
I20260927 03:54:07.517524 6828 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:07.517544 6828 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:07.517571 6828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:07.517757 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6760
I20260927 03:54:07.517845 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:07.518000 1225 external_mini_cluster.cc:1428] Setting key abd985aa298d81b04c18f1c8072c7950
I20260927 03:54:07.518170 6828 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } }
I20260927 03:54:07.518283 6828 leader_election.cc:302] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a6f214c9d941d3a383f44ac28e8aaf; no voters:
I20260927 03:54:07.518463 6828 leader_election.cc:288] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:07.518515 6831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:07.518745 6831 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [term 1 LEADER]: Becoming Leader. State: Replica: a3a6f214c9d941d3a383f44ac28e8aaf, State: Running, Role: LEADER
I20260927 03:54:07.518779 6828 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:07.518898 6831 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } }
I20260927 03:54:07.519294 6833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a6f214c9d941d3a383f44ac28e8aaf. Latest consensus state: current_term: 1 leader_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } } }
I20260927 03:54:07.519373 6833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:07.519578 6834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a6f214c9d941d3a383f44ac28e8aaf" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33587 } } }
I20260927 03:54:07.519691 6834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:07.519974 6842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:07.520432 6842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:07.521957 6842 catalog_manager.cc:1347] Generated new cluster ID: c0e732ff97404054bcf48decda7fa4fa
I20260927 03:54:07.522001 6842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:07.529374 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:07.518875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.1.50.126:33587)
0927 03:54:07.519094 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:07.519103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:07.519136 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:54:07.519271 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:07.519279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:07.519339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:07.520350 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:07.520441 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.521383 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.522518 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.523114 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.523211 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:07.525881 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:07.525886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:07.525887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:07.525899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:07.527800 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.528162 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.528166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.528166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.528292 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.528461 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.528461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.528462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.528780 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:07.528938 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:07.529006 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:54:07.529145 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260927 03:54:07.537809 6842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:07.538348 6842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:07.542273 6842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a6f214c9d941d3a383f44ac28e8aaf: Generated new TSK 0
I20260927 03:54:07.542428 6842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:54:07.570410 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:07.649472 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:07.649696 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:07.649729 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:07.650946 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:07.650995 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:07.651032 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:07.652525 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:07.652756 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:07.652966 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:07.657992 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.657994 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.657994 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:07.658216 6854 server_base.cc:1034] running on GCE node
I20260927 03:54:07.658483 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:07.658710 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:07.659864 6854 hybrid_clock.cc:648] HybridClock initialized: now 1790481247659847 us; error 29 us; skew 500 ppm
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:07.662477 6854 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:07.662971 6854 webserver.cc:460] Webserver started at http://127.1.50.65:43891/ using document root <none> and password file <none>
I20260927 03:54:07.663183 6854 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:07.663231 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:07.663348 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:07.664376 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "6e8d659639304efe86abf3a8f0368abd"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "8c09f6b574a7e8ed0e5b1a70478fb7ab"
server_key_iv: "20c48d173bdd700e1b0db1afca0ab50e"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.664705 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "6e8d659639304efe86abf3a8f0368abd"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "8c09f6b574a7e8ed0e5b1a70478fb7ab"
server_key_iv: "20c48d173bdd700e1b0db1afca0ab50e"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.665905 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:07.666689 6869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.666862 6854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:07.666944 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "6e8d659639304efe86abf3a8f0368abd"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "8c09f6b574a7e8ed0e5b1a70478fb7ab"
server_key_iv: "20c48d173bdd700e1b0db1afca0ab50e"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.667003 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:07.686833 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:07.688935 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:07.689080 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:07.689311 6854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:07.689610 6854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:07.689644 6854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.689673 6854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:07.689695 6854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.695026 6854 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:34983
I20260927 03:54:07.695078 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:34983 every 8 connection(s)
I20260927 03:54:07.695382 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:07.704087 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:07.696033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:42661 (local address 127.1.50.126:33587)
0927 03:54:07.696100 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:07.696102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:07.696282 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:54:07.697246 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:07.697248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:07.697259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:07.697319 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:07.697890 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.698422 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.699398 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.699593 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.699647 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:07.701673 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:07.701675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:07.701675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:07.701688 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:07.702998 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.703334 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.703334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.703335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.703377 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.703545 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.703546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.703546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.703635 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:07.703717 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:07.703953 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 03:54:07.704023 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:54:07.704170 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:07.696002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:33587 (local address 127.1.50.65:42661)
0927 03:54:07.696160 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:07.696202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:07.697099 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:07.697388 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:07.697392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:07.697632 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:07.697820 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:07.697825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.698499 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.698500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:07.699321 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:07.699323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.699651 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.699652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:07.699743 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:07.700158 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:07.700167 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:07.701569 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:07.703069 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:07.703074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:07.703076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:07.703279 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:07.703420 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:07.703421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:07.703421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:07.703505 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:07.703676 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:07.703679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:07.703803 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 03:54:07.703885 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:54:07.703998 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260927 03:54:07.704701 6983 heartbeater.cc:347] Connected to a master server at 127.1.50.126:33587
I20260927 03:54:07.704818 6983 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:07.704977 6983 heartbeater.cc:511] Master 127.1.50.126:33587 requested a full tablet report, sending...
I20260927 03:54:07.705367 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6854
I20260927 03:54:07.705457 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:07.705637 6792 ts_manager.cc:194] Registered new tserver with Master: 6e8d659639304efe86abf3a8f0368abd (127.1.50.65:34983)
I20260927 03:54:07.705662 1225 external_mini_cluster.cc:1428] Setting key a623dc9f5e8dc2c72471305a6da59d81
I20260927 03:54:07.706523 6792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:42661
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:07.733103 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:07.812373 6988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:07.812589 6988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:07.812613 6988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:07.813776 6988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:07.813817 6988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:07.813843 6988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:07.815341 6988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:07.815555 6988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:07.815771 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:54:07.820892 6988 server_base.cc:1034] running on GCE node
W20260927 03:54:07.820923 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.821039 6994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.821177 6993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:07.821420 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:07.821653 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:07.822800 6988 hybrid_clock.cc:648] HybridClock initialized: now 1790481247822786 us; error 32 us; skew 500 ppm
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:07.825342 6988 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:07.825838 6988 webserver.cc:460] Webserver started at http://127.1.50.66:39289/ using document root <none> and password file <none>
I20260927 03:54:07.826069 6988 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:07.826126 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:07.826251 6988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:07.827255 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "7cd6681ae1d94486943b297dcd468516"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "3c8f624559dac631b9479a256d703e3f"
server_key_iv: "58227d00cecd3051e6c563fcdd678a14"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.827608 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd6681ae1d94486943b297dcd468516"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "3c8f624559dac631b9479a256d703e3f"
server_key_iv: "58227d00cecd3051e6c563fcdd678a14"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.828897 6988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:07.829627 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.829774 6988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:07.829855 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "7cd6681ae1d94486943b297dcd468516"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "3c8f624559dac631b9479a256d703e3f"
server_key_iv: "58227d00cecd3051e6c563fcdd678a14"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.829921 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:07.843488 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:07.845443 6988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:07.845599 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:07.845844 6988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:07.846187 6988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:07.846222 6988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.846254 6988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:07.846271 6988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.851714 6988 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:41513
I20260927 03:54:07.851758 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:41513 every 8 connection(s)
I20260927 03:54:07.852053 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:07.857923 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 6988
I20260927 03:54:07.858036 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:07.858225 1225 external_mini_cluster.cc:1428] Setting key 16a5486f73f0ec1b936db00f475a1415
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:07.861655 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:07.852833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:36519 (local address 127.1.50.126:33587)
0927 03:54:07.852887 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:07.852889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:07.852995 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:54:07.853894 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:07.853894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:07.853904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:07.853968 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:07.854735 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.855251 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.856320 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.856510 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.856567 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:07.858992 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:07.858993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:07.858993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:07.858999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:07.860326 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.860730 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.860731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.860732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.860791 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:07.861014 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:07.861015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:07.861015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:07.861146 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:07.861204 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:07.861471 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:54:07.861552 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:54:07.861783 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:07.852721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:33587 (local address 127.1.50.66:36519)
0927 03:54:07.852878 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:07.852918 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:07.853809 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:07.854092 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:07.854097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:07.854434 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:07.854648 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:07.854653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.855370 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.855371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:07.856228 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:07.856230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:07.856631 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:07.856632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:07.856757 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:07.857256 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:07.857264 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:07.858892 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:07.860419 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:07.860425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:07.860427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:07.860663 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:07.860848 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:07.860848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:07.860848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:07.860962 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:07.861204 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:07.861207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:07.861328 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:54:07.861421 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:54:07.861572 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260927 03:54:07.862222 7117 heartbeater.cc:347] Connected to a master server at 127.1.50.126:33587
I20260927 03:54:07.862311 7117 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:07.862488 7117 heartbeater.cc:511] Master 127.1.50.126:33587 requested a full tablet report, sending...
I20260927 03:54:07.862846 6792 ts_manager.cc:194] Registered new tserver with Master: 7cd6681ae1d94486943b297dcd468516 (127.1.50.66:41513)
I20260927 03:54:07.863373 6792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:36519
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:07.885921 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:07.965080 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:07.965312 7122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:07.965337 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:07.966495 7122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:07.966528 7122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:07.966553 7122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:07.968005 7122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:07.968194 7122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:07.968389 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:07.973189 7128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.973229 7130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:07.973201 7127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:07.973433 7122 server_base.cc:1034] running on GCE node
I20260927 03:54:07.973639 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:07.973873 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:07.975031 7122 hybrid_clock.cc:648] HybridClock initialized: now 1790481247975021 us; error 37 us; skew 500 ppm
Sep 27 03:54:07 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:07.977465 7122 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:07.977942 7122 webserver.cc:460] Webserver started at http://127.1.50.67:36457/ using document root <none> and password file <none>
I20260927 03:54:07.978184 7122 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:07.978233 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:07.978346 7122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:07.979346 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "10d5fbd159aa41ea9538cb34bc04d90c"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "49c9dc45793eb45559f39193d02ee107"
server_key_iv: "94e25aa632995a68916b2728cc223090"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.979629 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "10d5fbd159aa41ea9538cb34bc04d90c"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "49c9dc45793eb45559f39193d02ee107"
server_key_iv: "94e25aa632995a68916b2728cc223090"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.980722 7122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:07.981355 7137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:07.981487 7122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:07.981559 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "10d5fbd159aa41ea9538cb34bc04d90c"
format_stamp: "Formatted at 2026-09-27 03:54:07 on dist-test-slave-96fg"
server_key: "49c9dc45793eb45559f39193d02ee107"
server_key_iv: "94e25aa632995a68916b2728cc223090"
server_key_version: "encryptionkey@0"
I20260927 03:54:07.981628 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:08.005679 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:08.007787 7122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:08.007916 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:08.008136 7122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:08.008422 7122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:08.008453 7122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:08.008476 7122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:08.008498 7122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:08.013814 7122 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:33547
I20260927 03:54:08.013856 7250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:33547 every 8 connection(s)
I20260927 03:54:08.014218 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:08.020934 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7122
I20260927 03:54:08.021023 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:08.021205 1225 external_mini_cluster.cc:1428] Setting key 63e3f66f53149e7f73d9bbb9fa04cb2d
I20260927 03:54:08.023874 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:08.014939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:52513 (local address 127.1.50.126:33587)
0927 03:54:08.014984 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.014987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.015207 (+ 220us) server_negotiation.cc:407] Connection header received
0927 03:54:08.016117 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.016118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.016129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.016178 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.016826 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.017447 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.018567 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.018813 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.018871 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:08.021249 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:08.021251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:08.021251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:08.021261 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:08.022679 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.023090 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.023091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.023092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.023156 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.023362 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.023362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.023362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.023471 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:08.023519 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:08.023738 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 03:54:08.023805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:54:08.024071 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:08.014881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:33587 (local address 127.1.50.67:52513)
0927 03:54:08.015052 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:08.015090 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:08.016024 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:08.016239 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:08.016243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:08.016498 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:08.016717 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:08.016722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.017543 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.017544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:08.018488 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:08.018491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.018894 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.018895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:08.019006 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:08.019596 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:08.019605 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:08.021127 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:08.022781 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:08.022788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:08.022791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:08.023029 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:08.023211 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:08.023212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:08.023212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:08.023315 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:08.023509 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:08.023512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:08.023633 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:54:08.023714 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:54:08.023842 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20260927 03:54:08.024492 7251 heartbeater.cc:347] Connected to a master server at 127.1.50.126:33587
I20260927 03:54:08.024581 7251 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:08.024745 7251 heartbeater.cc:511] Master 127.1.50.126:33587 requested a full tablet report, sending...
I20260927 03:54:08.025048 6792 ts_manager.cc:194] Registered new tserver with Master: 10d5fbd159aa41ea9538cb34bc04d90c (127.1.50.67:33547)
I20260927 03:54:08.025416 6792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:52513
I20260927 03:54:08.033070 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/test-admin.keytab.
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.046823 1225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 03:54:08.048374 7258 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:08.047610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.1.50.65:34983)
0927 03:54:08.047820 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.047824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.047834 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:08.047858 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.047860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.047874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.047944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.048292 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35362: BlockingRecv error: recv got EOF from 127.0.0.1:35362 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":133,"threads_started":1}
W20260927 03:54:08.048434 1225 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.049405 1225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.050297 1225 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:54:08.057391 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:08.050774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.1.50.65:34983)
0927 03:54:08.050830 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.050832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.050877 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:54:08.050922 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.050922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.050932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.050975 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.051315 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.052006 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.052733 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.053061 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.053119 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:08.055051 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:08.055058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:08.055058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:08.055064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:08.056452 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.056768 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.056771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.056771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.056817 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.056961 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.056962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.056962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.057066 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:08.057156 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:08.057234 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:54:08.057291 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 03:54:08.059262 7258 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:08.058734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.1.50.65:34983)
0927 03:54:08.058797 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.058798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.058808 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:08.058825 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.058825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.058835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.058886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.059215 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35372: BlockingRecv error: recv got EOF from 127.0.0.1:35372 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 03:54:08.059391 1225 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.060303 1225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.061117 1225 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:54:08.068259 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:08.061621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.1.50.65:34983)
0927 03:54:08.061674 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.061676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.061732 (+ 56us) server_negotiation.cc:407] Connection header received
0927 03:54:08.061783 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.061784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.061792 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.061834 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.062304 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.062849 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.063647 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.063884 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.063952 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:08.065978 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:08.065979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:08.065979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:08.065985 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:08.067238 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.067531 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.067532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.067532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.067576 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.067769 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.067770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.067770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.067891 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:08.067967 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:08.068117 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:54:08.068174 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 03:54:08.069825 1225 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481222547165-1225-0/krb5kdc/krb5cc' of type FILE
W20260927 03:54:08.069854 7258 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:08.069268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.1.50.65:34983)
0927 03:54:08.069313 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.069314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.069358 (+ 44us) server_negotiation.cc:407] Connection header received
0927 03:54:08.069422 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.069423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.069431 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.069470 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.069810 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35382: BlockingRecv error: recv got EOF from 127.0.0.1:35382 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.070775 1225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:08.071547 1225 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:54:08.078428 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:08.072043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.1.50.65:34983)
0927 03:54:08.072077 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:08.072080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:08.072101 (+ 21us) server_negotiation.cc:407] Connection header received
0927 03:54:08.072148 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:08.072149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:08.072155 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:08.072204 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:08.072499 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.072988 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.073782 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:08.073971 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:08.074064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:08.076178 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:08.076179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:08.076179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:08.076185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:08.077378 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.077734 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.077735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.077735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.077780 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:08.077919 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:08.077919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:08.077919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:08.078005 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:08.078117 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:08.078292 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:54:08.078359 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:54:08.079038 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6854
I20260927 03:54:08.082839 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6988
I20260927 03:54:08.086489 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7122
I20260927 03:54:08.090169 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 6760
2026-09-27T03:54:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1787 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[7271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[7271](info): set up 2 sockets
Sep 27 03:54:08 dist-test-slave-96fg krb5kdc[7271](info): commencing operation
krb5kdc: starting...
W20260927 03:54:09.045964 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab.
Entry for principal oryx/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab.
I20260927 03:54:09.161118 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37409
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab
--principal=oryx/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:09.243415 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:09.243628 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:09.243649 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:09.244787 7284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:09.244825 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:09.244838 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:09.244850 7284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:09.244860 7284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:09.246356 7284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab
--principal=oryx/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:09.246565 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:09.246783 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:09.251492 7292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.251500 7289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.251504 7290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.251534 7284 server_base.cc:1034] running on GCE node
I20260927 03:54:09.251891 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:09.252162 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:09.253309 7284 hybrid_clock.cc:648] HybridClock initialized: now 1790481249253295 us; error 30 us; skew 500 ppm
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:09.255710 7284 init.cc:377] Logged in from keytab as oryx/127.1.50.126@KRBTEST.COM (short username oryx)
I20260927 03:54:09.256112 7284 webserver.cc:460] Webserver started at http://127.1.50.126:44695/ using document root <none> and password file <none>
I20260927 03:54:09.256307 7284 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:09.256345 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:09.256438 7284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:09.257472 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "11715ff2510f4dbcb863ef34eb22f8a3"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "012c07cbd00642223284417e8776935f"
server_key_iv: "0d2c06b4324fd6ca92d53c27eb60a56a"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.257755 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "11715ff2510f4dbcb863ef34eb22f8a3"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "012c07cbd00642223284417e8776935f"
server_key_iv: "0d2c06b4324fd6ca92d53c27eb60a56a"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.259006 7284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:09.259739 7299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.259884 7284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:09.259965 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "11715ff2510f4dbcb863ef34eb22f8a3"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "012c07cbd00642223284417e8776935f"
server_key_iv: "0d2c06b4324fd6ca92d53c27eb60a56a"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.260025 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:09.271030 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:09.273272 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:09.273413 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:09.277199 7284 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:37409
I20260927 03:54:09.277297 7351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:37409 every 8 connection(s)
I20260927 03:54:09.277570 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:09.278663 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:09.280932 7352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Bootstrap starting.
I20260927 03:54:09.281503 7352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:09.281766 7352 log.cc:824] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:09.282526 7352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: No bootstrap required, opened a new log
I20260927 03:54:09.284089 7352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } }
I20260927 03:54:09.284226 7352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:09.284250 7352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11715ff2510f4dbcb863ef34eb22f8a3, State: Initialized, Role: FOLLOWER
I20260927 03:54:09.284396 7352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } }
I20260927 03:54:09.284461 7352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:09.284482 7352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:09.284509 7352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:09.285293 7352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } }
I20260927 03:54:09.285379 7352 leader_election.cc:302] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11715ff2510f4dbcb863ef34eb22f8a3; no voters:
I20260927 03:54:09.285542 7352 leader_election.cc:288] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:09.285667 7355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:09.285772 7352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:09.285821 7355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 11715ff2510f4dbcb863ef34eb22f8a3, State: Running, Role: LEADER
I20260927 03:54:09.285903 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7284
I20260927 03:54:09.285905 7355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } }
I20260927 03:54:09.285986 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:09.286192 1225 external_mini_cluster.cc:1428] Setting key 2b062de1fa2c680818ae6b54ad5cb975
I20260927 03:54:09.286309 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11715ff2510f4dbcb863ef34eb22f8a3. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } }
I20260927 03:54:09.286386 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:09.286525 7356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } }
I20260927 03:54:09.286569 7356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:09.286811 7364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:09.287328 7364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:09.289537 7364 catalog_manager.cc:1347] Generated new cluster ID: 4c6e834d1a7a4b3da70d4290f1eaf272
I20260927 03:54:09.289593 7364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:09.297554 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.286645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.1.50.126:37409)
0927 03:54:09.287166 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.287176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.287210 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:54:09.287331 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.287338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.287394 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.287529 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.288229 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.289404 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.290454 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.291130 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.291241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.294011 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.294040 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.294040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.294053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.295946 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.296363 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.296365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.296366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.296465 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.296647 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.296647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.296647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.296949 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.297091 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.297196 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.297329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":69,"threads_started":1}
I20260927 03:54:09.317453 7364 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:09.317987 7364 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:09.330047 7364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Generated new TSK 0
I20260927 03:54:09.330231 7364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
I20260927 03:54:09.364769 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:09.465008 7378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:09.465229 7378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:09.465261 7378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:09.466475 7378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:09.466516 7378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:09.466550 7378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:09.467988 7378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:09.468225 7378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:09.468434 7378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:54:09.473570 7378 server_base.cc:1034] running on GCE node
W20260927 03:54:09.473538 7383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.473539 7386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.473618 7384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.473935 7378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:09.474211 7378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:09.475364 7378 hybrid_clock.cc:648] HybridClock initialized: now 1790481249475345 us; error 32 us; skew 500 ppm
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:09.477800 7378 init.cc:377] Logged in from keytab as oryx/127.1.50.65@KRBTEST.COM (short username oryx)
I20260927 03:54:09.478291 7378 webserver.cc:460] Webserver started at http://127.1.50.65:41729/ using document root <none> and password file <none>
I20260927 03:54:09.478502 7378 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:09.478545 7378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:09.478649 7378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:09.479660 7378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "10ec0ea6482b13e6aad8b5f0a738f74d"
server_key_iv: "35cdede670bae44dcfdd640eff2f0698"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.479967 7378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "10ec0ea6482b13e6aad8b5f0a738f74d"
server_key_iv: "35cdede670bae44dcfdd640eff2f0698"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.481226 7378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:09.481941 7393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.482100 7378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:09.482183 7378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "10ec0ea6482b13e6aad8b5f0a738f74d"
server_key_iv: "35cdede670bae44dcfdd640eff2f0698"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.482241 7378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:09.494616 7378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:09.496608 7378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:09.496744 7378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:09.496951 7378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:09.497241 7378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:09.497273 7378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.497303 7378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:09.497319 7378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.502861 7378 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:46253
I20260927 03:54:09.502900 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:46253 every 8 connection(s)
I20260927 03:54:09.503204 7378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:09.510193 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7378
I20260927 03:54:09.510309 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:09.510492 1225 external_mini_cluster.cc:1428] Setting key 3ac6248c620139cc80f29fda8d12dd67
I20260927 03:54:09.512823 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.503894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44441 (local address 127.1.50.126:37409)
0927 03:54:09.503939 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.503941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.504143 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:54:09.504585 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.504586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.504596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.504704 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.505548 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.506161 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.507142 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.507350 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.507405 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.509655 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.509656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.509657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.509663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.511341 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.511807 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.511808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.511809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.511863 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.512134 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.512135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.512136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.512247 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.512300 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.512627 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.512730 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:54:09.512954 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.503866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:44441)
0927 03:54:09.504015 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:09.504062 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:09.504511 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:09.504715 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:09.504718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:09.505269 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:09.505457 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.505461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.506242 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.506243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.507068 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.507070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.507441 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.507442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.507571 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.508093 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:09.508100 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:09.509544 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:09.511421 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.511427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.511429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.511709 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.511938 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.511939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.511939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.512073 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.512303 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:09.512308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:09.512465 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 03:54:09.512576 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:54:09.512750 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260927 03:54:09.513381 7507 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:09.513489 7507 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:09.513675 7507 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:09.514307 7316 ts_manager.cc:194] Registered new tserver with Master: cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
I20260927 03:54:09.515221 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:44441
WARNING: no policy specified for oryx/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
I20260927 03:54:09.534840 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:09.614732 7512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:09.614938 7512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:09.614960 7512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:09.616142 7512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:09.616194 7512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:09.616219 7512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:09.617620 7512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:09.617838 7512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:09.618070 7512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:09.622928 7517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.622946 7518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.623044 7512 server_base.cc:1034] running on GCE node
W20260927 03:54:09.622933 7520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.623241 7512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:09.623443 7512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:09.624591 7512 hybrid_clock.cc:648] HybridClock initialized: now 1790481249624578 us; error 29 us; skew 500 ppm
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:09.627327 7512 init.cc:377] Logged in from keytab as oryx/127.1.50.66@KRBTEST.COM (short username oryx)
I20260927 03:54:09.627835 7512 webserver.cc:460] Webserver started at http://127.1.50.66:45373/ using document root <none> and password file <none>
I20260927 03:54:09.628054 7512 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:09.628124 7512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:09.628252 7512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:09.629329 7512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "cdc956cd6be41154de624d8efa3f0ef1"
server_key_iv: "1ee1514f8dbf9ea0f0457eb482925f0f"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.629668 7512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "cdc956cd6be41154de624d8efa3f0ef1"
server_key_iv: "1ee1514f8dbf9ea0f0457eb482925f0f"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.630987 7512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:09.631769 7527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.631949 7512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:09.632031 7512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "cdc956cd6be41154de624d8efa3f0ef1"
server_key_iv: "1ee1514f8dbf9ea0f0457eb482925f0f"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.632104 7512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:09.656229 7512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:09.658407 7512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:09.658545 7512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:09.658778 7512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:09.659075 7512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:09.659117 7512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.659147 7512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:09.659170 7512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.664825 7512 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:36331
I20260927 03:54:09.664871 7640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:36331 every 8 connection(s)
I20260927 03:54:09.665179 7512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:09.670197 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7512
I20260927 03:54:09.670305 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:09.670472 1225 external_mini_cluster.cc:1428] Setting key e7e37ce741ce3b7ef44867a4d01524db
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:09.674417 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.665824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:55665 (local address 127.1.50.126:37409)
0927 03:54:09.665871 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.665874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.666065 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:54:09.666577 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.666579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.666591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.666639 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.667629 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.668123 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.669087 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.669277 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.669334 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.671551 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.671553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.671553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.671561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.672877 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.673386 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.673387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.673388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.673444 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.673734 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.673735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.673736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.673854 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.673918 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.674266 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.674348 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:54:09.674635 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.665801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:55665)
0927 03:54:09.665940 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:09.665976 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:09.666483 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:09.666753 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:09.666757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:09.667314 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:09.667532 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.667537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.668204 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.668206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.669008 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.669010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.669346 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.669347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.669464 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.669999 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:09.670007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:09.671465 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:09.672953 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.672960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.672962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.673288 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.673510 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.673511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.673511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.673638 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.673918 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:09.673923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:09.674106 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 03:54:09.674238 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 03:54:09.674407 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260927 03:54:09.675220 7641 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:09.675341 7641 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:09.676476 7641 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:09.676920 7316 ts_manager.cc:194] Registered new tserver with Master: c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331)
I20260927 03:54:09.677438 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:55665
WARNING: no policy specified for oryx/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
I20260927 03:54:09.697763 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:09.776420 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:09.776634 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:09.776667 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:09.777781 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:09.777820 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:09.777845 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:09.779320 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:09.779516 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:09.779711 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:09.784492 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.784574 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.784466 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.784646 7646 server_base.cc:1034] running on GCE node
I20260927 03:54:09.784885 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:09.785122 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:09.786260 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790481249786237 us; error 49 us; skew 500 ppm
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:09.788923 7646 init.cc:377] Logged in from keytab as oryx/127.1.50.67@KRBTEST.COM (short username oryx)
I20260927 03:54:09.789340 7646 webserver.cc:460] Webserver started at http://127.1.50.67:41787/ using document root <none> and password file <none>
I20260927 03:54:09.789528 7646 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:09.789568 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:09.789659 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:09.790740 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "a0f32d5414c34a6490f550a012ec36c1"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "69842f95115a907017a4f052d3394678"
server_key_iv: "d7e0ef9537cd9d419ceb78613fab2bde"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.791026 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "a0f32d5414c34a6490f550a012ec36c1"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "69842f95115a907017a4f052d3394678"
server_key_iv: "d7e0ef9537cd9d419ceb78613fab2bde"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.792184 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:09.792968 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.793159 7646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:09.793241 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "a0f32d5414c34a6490f550a012ec36c1"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "69842f95115a907017a4f052d3394678"
server_key_iv: "d7e0ef9537cd9d419ceb78613fab2bde"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.793296 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:09.812211 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:09.814240 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:09.814356 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:09.814539 7646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:09.814841 7646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:09.814874 7646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.814903 7646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:09.814929 7646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.820281 7646 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42101
I20260927 03:54:09.820326 7774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42101 every 8 connection(s)
I20260927 03:54:09.820688 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:09.822418 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7646
I20260927 03:54:09.822486 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:09.822620 1225 external_mini_cluster.cc:1428] Setting key 43ae05bf3b70ba5a3d8eda78f9136c52
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:09.829730 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.821346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:36601 (local address 127.1.50.126:37409)
0927 03:54:09.821419 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.821422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.821593 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:54:09.822113 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.822114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.822128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.822182 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.823197 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.823775 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.824764 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.824944 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.824997 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.827095 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.827096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.827097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.827116 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.828355 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.828743 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.828744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.828744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.828803 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.829080 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.829081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.829081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.829217 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.829280 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.829579 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.829644 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:54:09.829928 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.821333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:36601)
0927 03:54:09.821483 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:09.821518 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:09.821985 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:09.822242 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:09.822246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:09.822885 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:09.823124 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.823132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.823864 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.823865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.824693 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.824695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.825036 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.825037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.825148 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.825652 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:09.825660 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:09.826996 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:09.828434 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.828439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.828441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.828661 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.828892 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.828893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.828893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.828997 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.829333 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:09.829339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:09.829461 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:54:09.829556 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:54:09.829682 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260927 03:54:09.830406 7775 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:09.830500 7775 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:09.830672 7775 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:09.831017 7316 ts_manager.cc:194] Registered new tserver with Master: a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:09.831388 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:36601
I20260927 03:54:09.834141 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.50.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.125.keytab.
Entry for principal oryx/127.1.50.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.125.keytab.
I20260927 03:54:09.861668 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:35635
--webserver_interface=127.1.50.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.125.keytab
--principal=oryx/127.1.50.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:37409,127.1.50.125:35635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:09.941092 7780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:09.941322 7780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:09.941344 7780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:09.942469 7780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:09.942504 7780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:09.942517 7780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:09.942528 7780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:09.942538 7780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:09.943979 7780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:37409,127.1.50.125:35635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.125.keytab
--principal=oryx/127.1.50.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.50.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
W20260927 03:54:09.944079 7780 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.50.126:37409,127.1.50.125:35635'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 03:54:09.944193 7780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:09.944365 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:09.949076 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.949129 7780 server_base.cc:1034] running on GCE node
W20260927 03:54:09.949136 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:09.949083 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:09.949555 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:09.949842 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:09.951071 7780 hybrid_clock.cc:648] HybridClock initialized: now 1790481249951049 us; error 43 us; skew 500 ppm
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:09.953418 7780 init.cc:377] Logged in from keytab as oryx/127.1.50.125@KRBTEST.COM (short username oryx)
I20260927 03:54:09.953859 7780 webserver.cc:460] Webserver started at http://127.1.50.125:38655/ using document root <none> and password file <none>
I20260927 03:54:09.954088 7780 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:09.954136 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:09.954259 7780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:09.955279 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/instance:
uuid: "0ee504b3d98541318bb7b10057405cf5"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "eb7d4dc582aba339d5c691c3213396a8"
server_key_iv: "463dcea020bbaf7eacd1320a3de73586"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.955608 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal/instance:
uuid: "0ee504b3d98541318bb7b10057405cf5"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "eb7d4dc582aba339d5c691c3213396a8"
server_key_iv: "463dcea020bbaf7eacd1320a3de73586"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.956918 7780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:09.957592 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:09.957775 7780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:09.957860 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "0ee504b3d98541318bb7b10057405cf5"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "eb7d4dc582aba339d5c691c3213396a8"
server_key_iv: "463dcea020bbaf7eacd1320a3de73586"
server_key_version: "encryptionkey@0"
I20260927 03:54:09.957922 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:09.975241 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:09.977371 7780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:09.977519 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:09.981153 7780 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.125:35635
I20260927 03:54:09.981204 7847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.125:35635 every 8 connection(s)
I20260927 03:54:09.981536 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
I20260927 03:54:09.982503 7848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:09.983963 7848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:54:09.986493 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7780
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.125@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:09.993646 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.984628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.1.50.126:37409)
0927 03:54:09.984696 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.984698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.984885 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:54:09.985332 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.985333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.985343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.985397 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.986057 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.986599 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.987987 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.988269 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.988342 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.990336 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.990337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.990338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.990344 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.992065 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.992544 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.992545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.992545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.992614 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.992910 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.992912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.992912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.993112 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.993216 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.993472 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.993547 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:54:09.993817 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.984601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44748)
0927 03:54:09.984758 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:09.984799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:09.985252 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:09.985449 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:09.985452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:09.985765 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:09.985956 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.985960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.986693 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.986694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.987905 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.987907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.988341 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.988342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.988445 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.988914 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:09.988922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:09.990252 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:09.992149 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.992154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.992155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.992461 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.992700 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:09.992701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:09.992701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:09.992833 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:09.993204 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:09.993209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:09.993350 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 03:54:09.993435 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:54:09.993611 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260927 03:54:09.994386 7848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:54:09.994895 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.986952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.1.50.125:35635)
0927 03:54:09.987226 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.987230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.987243 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:54:09.987258 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.987259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.987272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.987324 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.987985 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.988775 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.989441 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.989751 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.989804 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.991959 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.991966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.991967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.991973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:09.993704 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.994086 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.994089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.994090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.994151 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:09.994340 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:09.994342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:09.994342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:09.994482 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:09.994610 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:09.994731 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:54:09.994803 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
Sep 27 03:54:09 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.125@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.000905 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.994606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41530)
0927 03:54:09.994644 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:09.994654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:09.994685 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:09.995033 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:09.995034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:09.995162 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:09.995208 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.995209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.995860 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.995861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.996394 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:09.996396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.996710 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.996711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:09.996766 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.997080 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:09.997084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:09.998289 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.000150 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.000150 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.000151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.000296 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.000450 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.000450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.000451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.000495 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.000664 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.000664 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.000696 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.000742 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.000814 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:54:10.000993 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:09.994638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.1.50.125:35635)
0927 03:54:09.994933 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:09.994935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:09.994942 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:09.994959 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:09.994959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:09.994969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:09.995031 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:09.995279 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.995788 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.996456 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:09.996654 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:09.996704 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:09.998360 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:09.998361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:09.998362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:09.998367 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.000069 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.000360 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.000360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.000361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.000405 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.000535 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.000535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.000536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.000620 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.000662 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.000758 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.000856 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":277,"thread_start_us":88,"threads_started":1}
I20260927 03:54:10.002336 7848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Bootstrap starting.
I20260927 03:54:10.002897 7848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:10.003139 7848 log.cc:824] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:10.003780 7848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: No bootstrap required, opened a new log
I20260927 03:54:10.005122 7848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } }
I20260927 03:54:10.005239 7848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:10.005259 7848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee504b3d98541318bb7b10057405cf5, State: Initialized, Role: FOLLOWER
I20260927 03:54:10.005366 7848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } }
I20260927 03:54:10.005612 7848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:10.005584 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } }
I20260927 03:54:10.005715 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:54:10.007083 7866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:10.007126 7866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:10.008744 7780 master_runner.cc:428] Detected that this master 0ee504b3d98541318bb7b10057405cf5 is joining an existing cluster
I20260927 03:54:10.008800 7780 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.50.125:35635
I20260927 03:54:10.016677 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.009889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.1.50.125:35635)
0927 03:54:10.009971 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.009973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.010040 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:54:10.010237 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.010237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.010246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.010296 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.010636 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.011389 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.012174 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.012402 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.012469 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.013526 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.013527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.013527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.013533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.015104 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.015763 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.015765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.015765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.015824 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.016019 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.016020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.016020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.016150 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.016217 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.016366 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.016554 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 03:54:10.016822 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.009800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41536)
0927 03:54:10.009959 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.009970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.010069 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.010306 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.010307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.010509 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.010560 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.010562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.011479 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.011480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.012079 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.012081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.012504 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.012506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.012574 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.013045 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.013050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.013448 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.015472 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.015473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.015474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.015685 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.015894 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.015894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.015895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.015957 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.016214 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.016215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.016258 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.016405 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.016605 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":61,"spinlock_wait_cycles":10496,"thread_start_us":97,"threads_started":1}
I20260927 03:54:10.017561 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.009799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44756)
0927 03:54:10.009959 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.009970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.010062 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.010210 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.010211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.010479 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.010560 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.010562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.012345 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.012346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.013058 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.013061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.013451 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.013452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.013515 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.013955 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.013959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.014349 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.016300 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.016301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.016301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.016527 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.016848 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.016849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.016849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.016984 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.017260 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.017260 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.017300 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.017363 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.017459 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260927 03:54:10.017556 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.009934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.1.50.126:37409)
0927 03:54:10.010002 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.010003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.010011 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:10.010133 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.010134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.010145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.010193 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.011494 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.012245 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.013134 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.013378 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.013432 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.014647 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.014648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.014649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.014656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.016230 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.016607 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.016609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.016609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.016666 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.017056 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.017057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.017057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.017177 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.017219 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.017406 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.017473 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.126@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.027078 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.019780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41546)
0927 03:54:10.019971 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.020023 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.020098 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.020226 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.020229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.020565 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.020626 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.020628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.021196 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.021197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.022173 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.022176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.022500 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.022501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.022593 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.022987 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.022992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.024499 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.025936 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.025945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.025947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.026261 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.026473 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.026473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.026474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.026527 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.026739 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.026746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.026796 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.026864 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.026978 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260927 03:54:10.027072 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.019813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.1.50.125:35635)
0927 03:54:10.019853 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.019855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.020084 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:54:10.020170 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.020170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.020178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.020220 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.020690 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.021131 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.022248 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.022431 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.022494 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.024595 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.024597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.024597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.024603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.025860 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.026358 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.026359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.026360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.026420 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.026579 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.026580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.026581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.026688 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.026741 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.026896 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.026974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:10.027621 7316 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.50.125:35635: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ee504b3d98541318bb7b10057405cf5"
member_type: NON_VOTER
last_known_addr {
host: "127.1.50.125"
port: 35635
}
attrs {
promote: true
}
}
dest_uuid: "11715ff2510f4dbcb863ef34eb22f8a3"
cas_config_opid_index: -1
I20260927 03:54:10.027912 7316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } }
I20260927 03:54:10.028354 7876 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.028602 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:10.029397 7876 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.50.125:35635
I20260927 03:54:10.029403 7822 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 03:54:10.029485 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.029544 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.029649 7780 master.cc:537] Master@127.1.50.125:35635 shutting down...
I20260927 03:54:10.029742 7875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:54:10.030414 7300 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 03:54:10.030750 7300 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 -> Peer 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125:35635): Couldn't send request to peer 0ee504b3d98541318bb7b10057405cf5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:54:10.032210 7780 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:10.032289 7780 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:10.032325 7780 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: stopping tablet replica
I20260927 03:54:10.045377 7780 master.cc:552] Master@127.1.50.125:35635 shutdown complete.
I20260927 03:54:10.045439 7780 master_runner.cc:305] Clearing existing system tablet
I20260927 03:54:10.045564 7780 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.045610 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.046463 7780 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:54:10.046940 7881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.047075 7780 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.047130 7780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.047152 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "0ee504b3d98541318bb7b10057405cf5"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "eb7d4dc582aba339d5c691c3213396a8"
server_key_iv: "463dcea020bbaf7eacd1320a3de73586"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.047823 7780 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:10.049204 7780 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:54:10.049258 7780 log.cc:1196] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 03:54:10.049417 7780 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Deleting consensus metadata
I20260927 03:54:10.049568 7780 master_runner.cc:315] Copying system tablet from 127.1.50.126:37409
I20260927 03:54:10.050170 7780 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: tablet copy: Beginning tablet copy session from remote peer at address 127.1.50.126:37409
I20260927 03:54:10.056284 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.050519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.1.50.126:37409)
0927 03:54:10.050563 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.050565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.050602 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:54:10.050680 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.050680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.050689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.050732 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.051104 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.051589 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.052667 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.052871 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.052926 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.053909 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.053910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.053911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.053917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.055184 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.055557 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.055558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.055558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.055627 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.055831 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.055833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.055833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.055951 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.056018 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.056148 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.056215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:54:10.056347 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.050435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44768)
0927 03:54:10.050536 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.050548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.050620 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.050740 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.050741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.050937 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.050998 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.051000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.051666 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.051667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.052560 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.052564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.052941 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.052942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.053006 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.053420 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.053425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.053820 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.055254 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.055255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.055256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.055469 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.055692 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.055693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.055694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.055755 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.056015 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.056016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.056061 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.056127 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.056233 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":57,"threads_started":1}
I20260927 03:54:10.056766 7336 tablet_copy_service.cc:140] P 11715ff2510f4dbcb863ef34eb22f8a3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ee504b3d98541318bb7b10057405cf5 ({username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:44768)
I20260927 03:54:10.056859 7336 tablet_copy_service.cc:161] P 11715ff2510f4dbcb863ef34eb22f8a3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ee504b3d98541318bb7b10057405cf5 at {username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:44768: session id = 0ee504b3d98541318bb7b10057405cf5-00000000000000000000000000000000
I20260927 03:54:10.057490 7336 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:54:10.058192 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:54:10.059113 7780 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: tablet copy: Starting download of 0 data blocks...
I20260927 03:54:10.059243 7780 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: tablet copy: Starting download of 1 WAL segments...
I20260927 03:54:10.060145 7780 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:54:10.060886 7336 tablet_copy_service.cc:342] P 11715ff2510f4dbcb863ef34eb22f8a3: Request end of tablet copy session 0ee504b3d98541318bb7b10057405cf5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:44768
I20260927 03:54:10.060932 7336 tablet_copy_service.cc:433] P 11715ff2510f4dbcb863ef34eb22f8a3: ending tablet copy session 0ee504b3d98541318bb7b10057405cf5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ee504b3d98541318bb7b10057405cf5
I20260927 03:54:10.063414 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.064533 7888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.064545 7891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.064697 7780 server_base.cc:1034] running on GCE node
W20260927 03:54:10.064769 7889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.064930 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.065050 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.066156 7780 hybrid_clock.cc:648] HybridClock initialized: now 1790481250066149 us; error 23 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.067330 7780 init.cc:377] Logged in from keytab as oryx/127.1.50.125@KRBTEST.COM (short username oryx)
I20260927 03:54:10.067550 7780 webserver.cc:460] Webserver started at http://127.1.50.125:35677/ using document root <none> and password file <none>
I20260927 03:54:10.067657 7780 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.067692 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.068219 7780 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:10.068666 7898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.068768 7780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.068806 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "0ee504b3d98541318bb7b10057405cf5"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "eb7d4dc582aba339d5c691c3213396a8"
server_key_iv: "463dcea020bbaf7eacd1320a3de73586"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.068859 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.091342 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.092754 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.095438 7780 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.125:35635
I20260927 03:54:10.095531 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.125:35635 every 8 connection(s)
I20260927 03:54:10.095723 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
I20260927 03:54:10.096566 7951 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:54:10.097167 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Bootstrap starting.
I20260927 03:54:10.099565 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:54:10.099798 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Bootstrap complete.
I20260927 03:54:10.099985 7951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } }
I20260927 03:54:10.100042 7951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ee504b3d98541318bb7b10057405cf5, State: Initialized, Role: LEARNER
I20260927 03:54:10.100075 7951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } }
I20260927 03:54:10.100241 7951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:10.100382 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.100555 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:54:10.102401 7966 catalog_manager.cc:1259] Loaded cluster ID: 4c6e834d1a7a4b3da70d4290f1eaf272
I20260927 03:54:10.102445 7966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: loading cluster ID for follower catalog manager: success
I20260927 03:54:10.103163 7966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: acquiring CA information for follower catalog manager: success
I20260927 03:54:10.103431 7966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:54:10.104228 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.096328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.1.50.125:35635)
0927 03:54:10.096470 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.096471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.096481 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:10.096505 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.096506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.096517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.096608 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.097259 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.098311 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.099259 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.099549 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.099735 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.101027 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.101028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.101029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.101036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.103167 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.103562 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.103563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.103564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.103627 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.103825 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.103826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.103826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.103915 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.103983 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.104100 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.104170 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.1.50.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.124.keytab.
Entry for principal oryx/127.1.50.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.124.keytab.
I20260927 03:54:10.133086 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:40027
--webserver_interface=127.1.50.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.124.keytab
--principal=oryx/127.1.50.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:10.215619 7969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:10.215833 7969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:10.215858 7969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:10.216984 7969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:10.217024 7969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:10.217036 7969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:10.217047 7969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:10.217058 7969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:10.218596 7969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.124.keytab
--principal=oryx/127.1.50.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.50.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:10.218801 7969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:10.218972 7969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.223606 7974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.223609 7977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.223865 7969 server_base.cc:1034] running on GCE node
W20260927 03:54:10.223600 7975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.224148 7969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.224422 7969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.225584 7969 hybrid_clock.cc:648] HybridClock initialized: now 1790481250225570 us; error 27 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.228307 7969 init.cc:377] Logged in from keytab as oryx/127.1.50.124@KRBTEST.COM (short username oryx)
I20260927 03:54:10.228853 7969 webserver.cc:460] Webserver started at http://127.1.50.124:37689/ using document root <none> and password file <none>
I20260927 03:54:10.229061 7969 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.229115 7969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.229246 7969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:10.230412 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/instance:
uuid: "08371f7789b04ca1bf27663f4117b638"
format_stamp: "Formatted at 2026-09-27 03:54:10 on dist-test-slave-96fg"
server_key: "564c18c25ddd16950dea565d15c18ae7"
server_key_iv: "a30eb0f5d6e57f0e4633b35bfcd309d5"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.230767 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal/instance:
uuid: "08371f7789b04ca1bf27663f4117b638"
format_stamp: "Formatted at 2026-09-27 03:54:10 on dist-test-slave-96fg"
server_key: "564c18c25ddd16950dea565d15c18ae7"
server_key_iv: "a30eb0f5d6e57f0e4633b35bfcd309d5"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.232177 7969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:10.233422 7984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.233605 7969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:10.233698 7969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "08371f7789b04ca1bf27663f4117b638"
format_stamp: "Formatted at 2026-09-27 03:54:10 on dist-test-slave-96fg"
server_key: "564c18c25ddd16950dea565d15c18ae7"
server_key_iv: "a30eb0f5d6e57f0e4633b35bfcd309d5"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.233770 7969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.260258 7969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.262176 7969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:10.262306 7969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.266763 7969 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.124:40027
I20260927 03:54:10.266809 8036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.124:40027 every 8 connection(s)
I20260927 03:54:10.267128 7969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
I20260927 03:54:10.268006 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:10.268246 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 7969
I20260927 03:54:10.270426 8037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.124@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:10.278177 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.271187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.1.50.126:37409)
0927 03:54:10.271257 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.271259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.271285 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:54:10.271306 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.271307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.271317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.271365 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.271876 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.272384 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.273201 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.273505 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.273579 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.275715 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.275716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.275716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.275724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.276836 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.277238 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.277240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.277240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.277302 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.277529 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.277530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.277530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.277674 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.277731 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.277984 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.278092 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:54:10.278352 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.270990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44784)
0927 03:54:10.271145 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.271188 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.271247 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.271410 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.271413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.271714 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.271778 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.271779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.272473 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.272474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.273122 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.273124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.273633 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.273634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.273693 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.274125 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.274133 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.275637 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.276909 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.276915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.276917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.277176 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.277364 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.277365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.277365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.277462 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.277720 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.277726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.277868 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.277953 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.278136 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260927 03:54:10.278982 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.268784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.1.50.124:40027)
0927 03:54:10.268942 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.268952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.268983 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:54:10.269658 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.269668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.269735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.269887 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.270493 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.271621 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.272312 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.272734 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.272830 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.275232 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.275238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.275238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.275245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.276896 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.277391 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.277394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.277394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.277471 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.277680 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.277681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.277681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.277901 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.278768 (+ 867us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.278815 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.278893 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260927 03:54:10.279868 8037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.124@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.286296 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.280086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41574)
0927 03:54:10.280126 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.280137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.280177 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.280319 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.280320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.280427 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.280473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.280475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.281198 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.281199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.281735 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.281736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.282146 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.282147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.282217 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.282598 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.282603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.283759 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.285299 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.285300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.285300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.285494 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.285709 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.285710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.285710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.285770 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.285979 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.285980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.286038 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.286098 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.286199 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:54:10.286296 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.280157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.1.50.125:35635)
0927 03:54:10.280219 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.280221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.280229 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:10.280248 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.280249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.280257 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.280301 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.280569 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.281073 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.281824 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.282061 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.282120 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.283841 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.283843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.283843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.283850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.285233 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.285549 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.285551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.285552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.285605 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.285822 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.285823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.285823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.285930 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.285982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.286123 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.286206 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:54:10.286864 8037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.124@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
I20260927 03:54:10.292608 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.287128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.0.0.1:40066)
0927 03:54:10.287161 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.287170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.287209 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.287319 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.287319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.287439 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.287463 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.287464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.288042 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.288043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.288502 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.288503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.288825 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.288826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.288867 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.289213 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.289217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.290424 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.291787 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.291788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.291788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.291925 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.292069 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.292070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.292070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.292117 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.292339 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.292339 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.292373 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.292434 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.292510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:54:10.293267 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.287162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.1.50.124:40027)
0927 03:54:10.287195 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.287196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.287204 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:10.287259 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.287260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.287268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.287316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.287518 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.287981 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.288563 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.288773 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.288824 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.290516 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.290517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.290518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.290524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.291721 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.291977 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.291978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.291979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.292022 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.292188 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.292189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.292189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.292289 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.292341 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.293089 (+ 748us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.293174 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:10.294560 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Bootstrap starting.
I20260927 03:54:10.295358 8037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:10.295629 8037 log.cc:824] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:10.296347 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: No bootstrap required, opened a new log
I20260927 03:54:10.297731 8037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } }
I20260927 03:54:10.297864 8037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:10.297885 8037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08371f7789b04ca1bf27663f4117b638, State: Initialized, Role: FOLLOWER
I20260927 03:54:10.297998 8037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } }
I20260927 03:54:10.298341 8037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:10.298311 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } } }
I20260927 03:54:10.298383 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:54:10.299937 8053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:54:10.299998 8053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:54:10.302139 7969 master_runner.cc:428] Detected that this master 08371f7789b04ca1bf27663f4117b638 is joining an existing cluster
I20260927 03:54:10.302184 7969 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.50.124:40027
I20260927 03:54:10.330560 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7378
I20260927 03:54:10.335059 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:46253
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:37409
--webserver_port=41729
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:54:10.340554 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41588)
0927 03:54:10.303295 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.303305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.303357 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.303503 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.303504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.303707 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.304226 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.304228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.304846 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.304847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.305433 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.305435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.305763 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.305764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.305823 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.306412 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.306415 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.306763 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.308786 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.308787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.308787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.338765 (+ 29978us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.339036 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.339036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.339036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.339100 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.340156 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.340158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.340227 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.340297 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.340426 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260927 03:54:10.341566 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.1.50.125:35635)
0927 03:54:10.303338 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.303340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.303349 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:10.303405 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.303406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.303416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.303459 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.304300 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.304779 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.305506 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.305690 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.305744 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.306851 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.306853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.306854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.306862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.308495 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.338895 (+ 30400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.338897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.338897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.338970 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.339597 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.339598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.339599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.339768 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.339841 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.341377 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.341471 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:54:10.341979 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.0.0.1:40076)
0927 03:54:10.303584 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.303593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.303651 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.303773 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.303773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.303976 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.304238 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.304240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.305098 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.305099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.305902 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.305904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.306289 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.306290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.306347 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.338556 (+ 32209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.338563 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.338941 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.341052 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.341053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.341053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.341232 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.341421 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.341421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.341421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.341465 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.341688 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.341689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.341736 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.341789 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.341878 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":74,"threads_started":1}
I20260927 03:54:10.342262 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.1.50.124:40027)
0927 03:54:10.303348 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.303350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.303654 (+ 304us) server_negotiation.cc:407] Connection header received
0927 03:54:10.303694 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.303695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.303704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.303756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.304326 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.305017 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.305989 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.306223 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.306291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.339032 (+ 32741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.339034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.339034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.339040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.340965 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.341296 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.341296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.341297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.341367 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.341514 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.341515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.341515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.341618 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.341670 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.342106 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.342188 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 03:54:10.343230 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44798)
0927 03:54:10.306091 (+ 2622us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.306100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.306186 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.306387 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.306388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.306581 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.306631 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.306632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.307404 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.307405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.308060 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.308062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.308421 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.308422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.308481 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.309155 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.309159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.339224 (+ 30065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.342194 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.342195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.342195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.342390 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.342603 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.342603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.342604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.342667 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.342934 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.342935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.342974 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.343020 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.343118 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"thread_start_us":67,"threads_started":1}
I20260927 03:54:10.343231 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.303508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.1.50.126:37409)
0927 03:54:10.303561 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.303564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.306157 (+ 2593us) server_negotiation.cc:407] Connection header received
0927 03:54:10.306261 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.306262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.306274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.306303 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.306718 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.307331 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.308137 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.308362 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.308417 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.339332 (+ 30915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.339334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.339335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.339343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.341252 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.342477 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.342479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.342479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.342537 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.342765 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.342766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.342766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.342877 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.342930 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.343080 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.343153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481249, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.126@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
I20260927 03:54:10.350906 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.345083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.0.0.1:40092)
0927 03:54:10.345125 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.345135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.345170 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.345370 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.345371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.345510 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.345550 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.345551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.346313 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.346315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.346825 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.346826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.347144 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.347145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.347214 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.347548 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.347553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.348653 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.349968 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.349969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.349969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.350157 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.350321 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.350323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.350323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.350376 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.350572 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.350573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.350658 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.350731 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.350829 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:54:10.351233 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.345227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.1.50.124:40027)
0927 03:54:10.345269 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.345270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.345277 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:10.345307 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.345307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.345315 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.345357 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.345609 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.346243 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.346891 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.347089 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.347142 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.348733 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.348734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.348735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.348741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.349897 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.350220 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.350221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.350222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.350270 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.350424 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.350425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.350425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.350523 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.350574 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.351082 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.351151 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:54:10.351711 7316 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.50.124:40027: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08371f7789b04ca1bf27663f4117b638"
member_type: NON_VOTER
last_known_addr {
host: "127.1.50.124"
port: 40027
}
attrs {
promote: true
}
}
dest_uuid: "11715ff2510f4dbcb863ef34eb22f8a3"
cas_config_opid_index: 5
I20260927 03:54:10.351938 7316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
I20260927 03:54:10.352250 7875 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08371f7789b04ca1bf27663f4117b638 (127.1.50.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.353230 7875 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.50.124:40027
I20260927 03:54:10.353305 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.353363 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.353756 8011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:10.354914 8011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:54:10.355172 7876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:10.355541 7969 master.cc:537] Master@127.1.50.124:40027 shutting down...
I20260927 03:54:10.356748 8011 raft_consensus.cc:858] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
W20260927 03:54:10.356832 8011 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
I20260927 03:54:10.357372 8011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.357887 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.352362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:41598)
0927 03:54:10.352594 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.352604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.352647 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.352842 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.352842 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.352844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.352881 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.352882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.353761 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.353762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.355137 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.355139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.357615 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.357616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.357680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.357720 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.357774 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.357820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 03:54:10.358199 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.352453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.1.50.125:35635)
0927 03:54:10.352566 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.352568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.352689 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:54:10.352721 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.352722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.352745 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.352793 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.352946 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.353600 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.355221 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.355815 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.355877 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.355957 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.358088 (+ 2131us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.358137 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 03:54:10.358741 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.358853 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:54:10.358893 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.358937 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:54:10.358973 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11715ff2510f4dbcb863ef34eb22f8a3. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } }
I20260927 03:54:10.359068 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:54:10.359217 7925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 LEARNER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:54:10.359411 7876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 03:54:10.359688 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.359771 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.360412 7925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08371f7789b04ca1bf27663f4117b638 (127.1.50.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
W20260927 03:54:10.361426 7301 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 -> Peer 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027): Couldn't send request to peer 08371f7789b04ca1bf27663f4117b638. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:54:10.361429 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11715ff2510f4dbcb863ef34eb22f8a3. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.361506 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:54:10.361543 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: NON_VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.361581 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:54:10.361647 7877 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: attempting to promote NON_VOTER 0ee504b3d98541318bb7b10057405cf5 to VOTER
I20260927 03:54:10.361800 7877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
W20260927 03:54:10.362310 7301 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 -> Peer 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027): Couldn't send request to peer 08371f7789b04ca1bf27663f4117b638. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:54:10.362370 7925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 LEARNER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 03:54:10.362521 7969 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 03:54:10.362541 7876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 03:54:10.362610 7969 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 03:54:10.362644 7969 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: stopping tablet replica
I20260927 03:54:10.363101 7876 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.363941 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.364012 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.364065 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.364110 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.364333 7925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.364979 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.365103 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:54:10.385260 7969 master.cc:552] Master@127.1.50.124:40027 shutdown complete.
I20260927 03:54:10.385313 7969 master_runner.cc:305] Clearing existing system tablet
I20260927 03:54:10.385438 7969 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.385480 7969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.386134 7969 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.386585 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.386767 7969 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.386818 7969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.386842 7969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "08371f7789b04ca1bf27663f4117b638"
format_stamp: "Formatted at 2026-09-27 03:54:10 on dist-test-slave-96fg"
server_key: "564c18c25ddd16950dea565d15c18ae7"
server_key_iv: "a30eb0f5d6e57f0e4633b35bfcd309d5"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.387485 7969 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:10.388832 7969 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:54:10.388892 7969 log.cc:1196] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 03:54:10.389124 7969 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Deleting consensus metadata
I20260927 03:54:10.389251 7969 master_runner.cc:315] Copying system tablet from 127.1.50.126:37409
I20260927 03:54:10.389788 7969 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: tablet copy: Beginning tablet copy session from remote peer at address 127.1.50.126:37409
I20260927 03:54:10.395254 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.390138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.1.50.126:37409)
0927 03:54:10.390186 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.390188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.390312 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:54:10.390372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.390373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.390383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.390427 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.390718 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.391198 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.391893 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.392087 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.392141 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.393018 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.393019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.393019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.393027 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.394289 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.394583 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.394584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.394584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.394636 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.394793 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.394793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.394794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.394900 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.394954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.395084 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.395160 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:54:10.395274 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.390154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:44802)
0927 03:54:10.390253 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.390262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.390324 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.390430 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.390431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.390593 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.390644 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.390645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.391274 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.391275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.391819 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.391821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.392146 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.392146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.392214 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.392626 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.392631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.392933 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.394361 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.394362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.394362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.394523 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.394688 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.394689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.394689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.394744 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.394951 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.394952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.394996 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.395056 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.395161 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":65,"threads_started":1}
I20260927 03:54:10.395525 7336 tablet_copy_service.cc:140] P 11715ff2510f4dbcb863ef34eb22f8a3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08371f7789b04ca1bf27663f4117b638 ({username='oryx', principal='oryx/127.1.50.124@KRBTEST.COM'} at 127.0.0.1:44802)
I20260927 03:54:10.395583 7336 tablet_copy_service.cc:161] P 11715ff2510f4dbcb863ef34eb22f8a3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08371f7789b04ca1bf27663f4117b638 at {username='oryx', principal='oryx/127.1.50.124@KRBTEST.COM'} at 127.0.0.1:44802: session id = 08371f7789b04ca1bf27663f4117b638-00000000000000000000000000000000
I20260927 03:54:10.396083 7336 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:54:10.396551 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:54:10.397583 7969 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: tablet copy: Starting download of 0 data blocks...
I20260927 03:54:10.397712 7969 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: tablet copy: Starting download of 1 WAL segments...
I20260927 03:54:10.398362 7969 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:54:10.399150 7336 tablet_copy_service.cc:342] P 11715ff2510f4dbcb863ef34eb22f8a3: Request end of tablet copy session 08371f7789b04ca1bf27663f4117b638-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.50.124@KRBTEST.COM'} at 127.0.0.1:44802
I20260927 03:54:10.399196 7336 tablet_copy_service.cc:433] P 11715ff2510f4dbcb863ef34eb22f8a3: ending tablet copy session 08371f7789b04ca1bf27663f4117b638-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08371f7789b04ca1bf27663f4117b638
I20260927 03:54:10.401557 7969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.402685 8080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.402844 7969 server_base.cc:1034] running on GCE node
W20260927 03:54:10.406952 8081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.407984 8083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.408154 7969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.408303 7969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.409426 7969 hybrid_clock.cc:648] HybridClock initialized: now 1790481250409436 us; error 46 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.410728 7969 init.cc:377] Logged in from keytab as oryx/127.1.50.124@KRBTEST.COM (short username oryx)
I20260927 03:54:10.410977 7969 webserver.cc:460] Webserver started at http://127.1.50.124:35693/ using document root <none> and password file <none>
I20260927 03:54:10.411091 7969 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.411125 7969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.411727 7969 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:10.412202 8090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.412299 7969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.412334 7969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "08371f7789b04ca1bf27663f4117b638"
format_stamp: "Formatted at 2026-09-27 03:54:10 on dist-test-slave-96fg"
server_key: "564c18c25ddd16950dea565d15c18ae7"
server_key_iv: "a30eb0f5d6e57f0e4633b35bfcd309d5"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.412392 7969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.421278 7969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.422894 7969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.425930 7969 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.124:40027
I20260927 03:54:10.425954 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.124:40027 every 8 connection(s)
I20260927 03:54:10.426297 7969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
I20260927 03:54:10.427029 8143 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:54:10.427423 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Bootstrap starting.
I20260927 03:54:10.429169 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:54:10.429352 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: Bootstrap complete.
I20260927 03:54:10.429555 8143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
I20260927 03:54:10.429627 8143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08371f7789b04ca1bf27663f4117b638, State: Initialized, Role: LEARNER
I20260927 03:54:10.429662 8143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } }
I20260927 03:54:10.429824 8143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:10.430102 8146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.430285 8146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:54:10.431610 8157 catalog_manager.cc:1259] Loaded cluster ID: 4c6e834d1a7a4b3da70d4290f1eaf272
I20260927 03:54:10.431689 8157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: loading cluster ID for follower catalog manager: success
I20260927 03:54:10.432439 8157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: acquiring CA information for follower catalog manager: success
I20260927 03:54:10.432696 8157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 03:54:10.456516 8062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:10.456844 8062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:10.456878 8062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:10.458755 8062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:10.458830 8062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:10.458873 8062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:10.461266 8062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:46253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=41729
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:10.461582 8062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:10.461838 8062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.467108 8162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.467242 8062 server_base.cc:1034] running on GCE node
W20260927 03:54:10.467096 8165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.467326 8163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.467554 8062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.467789 8062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.468941 8062 hybrid_clock.cc:648] HybridClock initialized: now 1790481250468925 us; error 35 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.471462 8062 init.cc:377] Logged in from keytab as oryx/127.1.50.65@KRBTEST.COM (short username oryx)
I20260927 03:54:10.471967 8062 webserver.cc:460] Webserver started at http://127.1.50.65:41729/ using document root <none> and password file <none>
I20260927 03:54:10.472170 8062 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.472231 8062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.473579 8062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:10.474208 8172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.474340 8062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:10.474422 8062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "10ec0ea6482b13e6aad8b5f0a738f74d"
server_key_iv: "35cdede670bae44dcfdd640eff2f0698"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.474799 8062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.488832 8062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.490769 8062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:10.490907 8062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.491106 8062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:10.491389 8062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:10.491421 8062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.491451 8062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:10.491472 8062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.497223 8062 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:46253
I20260927 03:54:10.497308 8285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:46253 every 8 connection(s)
I20260927 03:54:10.497605 8062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:10.500526 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8062
I20260927 03:54:10.500671 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7512
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.508912 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:54947 (local address 127.1.50.125:35635)
0927 03:54:10.498608 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.498610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.499884 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 03:54:10.499916 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.499917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.499928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.499978 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.500240 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.501000 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502141 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502433 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502470 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.506094 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.506096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.506097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.506103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.507712 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508040 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.508041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.508041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.508087 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508265 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.508265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.508266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.508385 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.508421 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.508743 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.508818 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:54:10.508921 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:51449 (local address 127.1.50.126:37409)
0927 03:54:10.498766 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.498768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.498777 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:10.499087 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.499088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.499100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.499142 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.499798 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.500530 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502125 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502401 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502441 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.506086 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.506087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.506087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.506095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.507840 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508195 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.508196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.508196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.508248 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508410 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.508411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.508411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.508525 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.508563 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.508757 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.508834 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:54:10.508944 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35183 (local address 127.1.50.124:40027)
0927 03:54:10.498731 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.498733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.498922 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:54:10.499205 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.499207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.499219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.499279 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.499797 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.500817 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502134 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502435 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502470 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.505489 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.505491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.505491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.505499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.507335 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.507819 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.507820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.507821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.507873 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508203 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.508204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.508204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.508340 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.508380 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.508743 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.508860 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":48,"threads_started":1}
I20260927 03:54:10.509512 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:51449)
0927 03:54:10.498468 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.498499 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.498999 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.499221 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.499225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.499492 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.499715 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.499719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.500753 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.500754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502012 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.502014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502522 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502706 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.503320 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.503330 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.505962 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.507924 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.507925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.507926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.508120 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.508307 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.508308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.508356 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.508601 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.508602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.508653 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.509034 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.509324 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":108,"thread_start_us":81,"threads_started":1}
I20260927 03:54:10.509518 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.1.50.65:35183)
0927 03:54:10.498850 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.498860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.499141 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.499294 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.499295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.499530 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.499715 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.499720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.500927 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.500928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502011 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.502033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502545 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502705 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.503492 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.503499 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.505403 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.507419 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.507425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.507427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.507746 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.507940 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.507941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.507941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.508131 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.508423 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.508429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.508590 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.509044 (+ 454us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.509253 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260927 03:54:10.510010 8286 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:10.510130 8286 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.510264 8287 heartbeater.cc:347] Connected to a master server at 127.1.50.124:40027
I20260927 03:54:10.510308 8286 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:10.510329 8287 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.510386 8287 heartbeater.cc:511] Master 127.1.50.124:40027 requested a full tablet report, sending...
I20260927 03:54:10.510550 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.498310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.1.50.65:54947)
0927 03:54:10.499825 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.499833 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.499878 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.499979 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.499980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.500134 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.500187 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.500188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.501269 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.501270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502063 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.502065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.502544 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.502545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.502705 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.503847 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.503851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.505981 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.507791 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.507792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.507792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.507966 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.508150 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.508151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.508151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.508201 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.508461 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.508461 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.508598 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.509467 (+ 869us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.510265 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"mutex_wait_us":419,"thread_start_us":64,"threads_started":1}
I20260927 03:54:10.510703 7316 ts_manager.cc:194] Re-registered known tserver with Master: cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
I20260927 03:54:10.510821 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:36331
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:37409
--webserver_port=45373
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:54:10.511134 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:51449
I20260927 03:54:10.511471 8288 heartbeater.cc:347] Connected to a master server at 127.1.50.125:35635
I20260927 03:54:10.511516 8288 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.511569 8288 heartbeater.cc:511] Master 127.1.50.125:35635 requested a full tablet report, sending...
I20260927 03:54:10.511581 8107 ts_manager.cc:194] Registered new tserver with Master: cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
I20260927 03:54:10.511869 7915 ts_manager.cc:194] Registered new tserver with Master: cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
W20260927 03:54:10.589972 8294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:10.590219 8294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:10.590251 8294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:10.591480 8294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:10.591524 8294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:10.591559 8294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:10.592999 8294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:36331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=45373
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:10.593226 8294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:10.593428 8294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.598387 8302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.598415 8300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.598503 8299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.598551 8294 server_base.cc:1034] running on GCE node
I20260927 03:54:10.598745 8294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.598992 8294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.600117 8294 hybrid_clock.cc:648] HybridClock initialized: now 1790481250600103 us; error 34 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.602686 8294 init.cc:377] Logged in from keytab as oryx/127.1.50.66@KRBTEST.COM (short username oryx)
I20260927 03:54:10.603071 8294 webserver.cc:460] Webserver started at http://127.1.50.66:45373/ using document root <none> and password file <none>
I20260927 03:54:10.603251 8294 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.603299 8294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.604524 8294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:10.605008 8309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.605171 8294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.605254 8294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "cdc956cd6be41154de624d8efa3f0ef1"
server_key_iv: "1ee1514f8dbf9ea0f0457eb482925f0f"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.605579 8294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.617223 8294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.619288 8294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:10.619401 8294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.619626 8294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:10.619926 8294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:10.619959 8294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.619980 8294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:10.619992 8294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.626206 8294 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:36331
I20260927 03:54:10.626267 8422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:36331 every 8 connection(s)
I20260927 03:54:10.626695 8294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
I20260927 03:54:10.635620 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8294
I20260927 03:54:10.635780 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7646
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.640232 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.627554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:57079 (local address 127.1.50.124:40027)
0927 03:54:10.627643 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.627645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.627899 (+ 254us) server_negotiation.cc:407] Connection header received
0927 03:54:10.629093 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.629094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.629106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.629164 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.630061 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.630763 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632220 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632509 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632553 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.637276 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.637278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.637278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.637286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.638732 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.639186 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.639187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.639187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.639242 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.639504 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.639505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.639505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.639640 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.639680 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.640052 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.640138 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 03:54:10.640687 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.627465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.1.50.66:57079)
0927 03:54:10.627724 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.627787 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.628984 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.629283 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.629288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.629647 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.629936 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.629942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.630901 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.630902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.632120 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.632123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632648 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.632795 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.633860 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.633864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.637179 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.638818 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.638826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.638829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.639107 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.639308 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.639308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.639309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.639430 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.639737 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.639743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.639909 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.640331 (+ 422us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.640510 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260927 03:54:10.641114 8424 heartbeater.cc:347] Connected to a master server at 127.1.50.124:40027
I20260927 03:54:10.641216 8424 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.641443 8424 heartbeater.cc:511] Master 127.1.50.124:40027 requested a full tablet report, sending...
I20260927 03:54:10.641911 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.627500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:56377 (local address 127.1.50.126:37409)
0927 03:54:10.627605 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.627607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.629230 (+ 1623us) server_negotiation.cc:407] Connection header received
0927 03:54:10.629283 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.629284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.629295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.629348 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.630074 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.630765 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632254 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632552 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632589 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.636821 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.636824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.636824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.636832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.640528 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.641090 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.641091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.641092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.641153 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.641356 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.641358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.641358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.641495 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.641561 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.641751 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.641829 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 03:54:10.641913 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.627465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:56377)
0927 03:54:10.629176 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.629186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.629236 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.629354 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.629354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.629646 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.629936 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.629942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.630907 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.630907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.632165 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.632167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632658 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.632910 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.633364 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.633374 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.636718 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.640666 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.640667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.640667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.640955 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.641220 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.641221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.641221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.641291 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.641592 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.641593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.641642 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.641711 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.641819 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"mutex_wait_us":68,"thread_start_us":87,"threads_started":1}
I20260927 03:54:10.642141 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:42101
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:37409
--webserver_port=41787
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:54:10.643579 8423 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:10.643628 8423 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.643721 8423 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:10.643931 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.630374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:33191 (local address 127.1.50.125:35635)
0927 03:54:10.630417 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.630419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.631250 (+ 831us) server_negotiation.cc:407] Connection header received
0927 03:54:10.631354 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.631355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.631367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.631395 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.631754 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632463 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.633407 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.633711 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.633781 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.640002 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.640004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.640004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.640012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.642771 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.643113 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.643115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.643115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.643176 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.643376 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.643376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.643377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.643509 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.643549 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.643757 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.643835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:54:10.644001 7316 ts_manager.cc:194] Re-registered known tserver with Master: c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331)
I20260927 03:54:10.644234 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.630191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.1.50.66:33191)
0927 03:54:10.631185 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.631195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.631284 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.631454 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.631455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.631615 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.631668 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.631670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.632542 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.632543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.633300 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.633303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.634067 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.634068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.634145 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.634529 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.634532 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.639894 (+ 5362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.642862 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.642863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.642863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.643025 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.643235 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.643236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.643236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.643299 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.643591 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.643591 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.643632 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.644028 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.644125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":60,"threads_started":1}
I20260927 03:54:10.644521 8107 ts_manager.cc:194] Registered new tserver with Master: c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331)
I20260927 03:54:10.644726 8425 heartbeater.cc:347] Connected to a master server at 127.1.50.125:35635
I20260927 03:54:10.644767 8425 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.644824 8425 heartbeater.cc:511] Master 127.1.50.125:35635 requested a full tablet report, sending...
I20260927 03:54:10.645083 7915 ts_manager.cc:194] Registered new tserver with Master: c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331)
I20260927 03:54:10.645282 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:56377
W20260927 03:54:10.727690 8430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:10.727918 8430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:10.727949 8430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:10.729135 8430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:10.729176 8430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:10.729211 8430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:10.730813 8430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:42101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=41787
--tserver_master_addrs=127.1.50.126:37409,127.1.50.125:35635,127.1.50.124:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:10.731034 8430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:10.731240 8430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:10.736119 8435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:10.736119 8438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.736349 8430 server_base.cc:1034] running on GCE node
W20260927 03:54:10.736260 8436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:10.736578 8430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:10.736781 8430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:10.737929 8430 hybrid_clock.cc:648] HybridClock initialized: now 1790481250737914 us; error 29 us; skew 500 ppm
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:10.740355 8430 init.cc:377] Logged in from keytab as oryx/127.1.50.67@KRBTEST.COM (short username oryx)
I20260927 03:54:10.740772 8430 webserver.cc:460] Webserver started at http://127.1.50.67:41787/ using document root <none> and password file <none>
I20260927 03:54:10.740962 8430 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:10.741020 8430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:10.742331 8430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:10.742835 8445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.742986 8430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.743062 8430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "a0f32d5414c34a6490f550a012ec36c1"
format_stamp: "Formatted at 2026-09-27 03:54:09 on dist-test-slave-96fg"
server_key: "69842f95115a907017a4f052d3394678"
server_key_iv: "d7e0ef9537cd9d419ceb78613fab2bde"
server_key_version: "encryptionkey@0"
I20260927 03:54:10.743413 8430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:10.750980 8430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:10.752810 8430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:10.752933 8430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:10.753116 8430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:10.753383 8430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:10.753413 8430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.753441 8430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:10.753465 8430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:10.758762 8430 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42101
I20260927 03:54:10.758823 8558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42101 every 8 connection(s)
I20260927 03:54:10.759119 8430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.124@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
Sep 27 03:54:10 dist-test-slave-96fg krb5kdc[7271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481250, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.125@KRBTEST.COM
I20260927 03:54:10.768221 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8430
I20260927 03:54:10.769286 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.759855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:54613 (local address 127.1.50.124:40027)
0927 03:54:10.759914 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.759916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.760066 (+ 150us) server_negotiation.cc:407] Connection header received
0927 03:54:10.760604 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.760605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.760617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.760646 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.761384 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.762129 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763546 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763830 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763882 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.765937 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.765939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.765940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.765948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.767565 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768061 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.768062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.768063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.768119 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768506 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.768507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.768507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.768672 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.768715 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.769095 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.769178 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:54:10.769611 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.760084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:40169 (local address 127.1.50.125:35635)
0927 03:54:10.760161 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.760162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.760171 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:10.760596 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.760597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.760609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.760673 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.761350 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.762150 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763044 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763299 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763375 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.767010 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.767011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.767012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.767019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.768519 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768842 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.768843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.768843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.768894 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.769082 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.769083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.769083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.769201 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.769242 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.769450 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.769529 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:54:10.769814 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.759796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.1.50.67:40169)
0927 03:54:10.759951 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.759990 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.760498 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.760739 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.760742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.761062 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.761256 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.761260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.762230 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.762231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.762971 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.762973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763435 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.763583 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.764192 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.764196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.766715 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.768594 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.768595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.768768 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.768955 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.768956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.769004 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.769289 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.769289 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.769329 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.769407 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.769584 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260927 03:54:10.770262 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.759796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.1.50.67:54613)
0927 03:54:10.759962 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.759992 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.760512 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.761081 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.761082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.761206 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.761278 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.761279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.762212 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.762213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.763080 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.763081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763894 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.763954 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.764297 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.764302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.765763 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.767669 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.767677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.767679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.767969 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.768232 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.768232 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.768233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.768425 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.768775 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.768781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.768951 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.769781 (+ 830us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.770162 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":71,"thread_start_us":81,"threads_started":1}
I20260927 03:54:10.770428 8561 heartbeater.cc:347] Connected to a master server at 127.1.50.125:35635
I20260927 03:54:10.770535 8561 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.770701 8560 heartbeater.cc:347] Connected to a master server at 127.1.50.124:40027
I20260927 03:54:10.770737 8561 heartbeater.cc:511] Master 127.1.50.125:35635 requested a full tablet report, sending...
I20260927 03:54:10.770746 8560 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.770844 8560 heartbeater.cc:511] Master 127.1.50.124:40027 requested a full tablet report, sending...
I20260927 03:54:10.771217 7915 ts_manager.cc:194] Registered new tserver with Master: a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:10.771216 8107 ts_manager.cc:194] Registered new tserver with Master: a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:10.773130 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.759796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:41457)
0927 03:54:10.760108 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.760116 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.760511 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.760757 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.760758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.761072 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:10.761258 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.761261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.761953 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.761954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.763075 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.763077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763503 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.763580 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.764114 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:10.764123 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:10.769829 (+ 5706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:10.771815 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.771816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.771817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.772007 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.772250 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:10.772251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:10.772251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:10.772342 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:10.772621 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:10.772622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:10.772666 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.772736 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.772840 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":186,"thread_start_us":54,"threads_started":1}
I20260927 03:54:10.773491 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.760032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:41457 (local address 127.1.50.126:37409)
0927 03:54:10.760128 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.760130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.760165 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:54:10.760637 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.760638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.760649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.760692 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.761373 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.761859 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763164 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.763436 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.763493 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.769911 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.769912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.769913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.769920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.771765 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.772066 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.772067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.772068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.772158 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.772412 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.772413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.772413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.772542 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.772595 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.773309 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.773386 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 03:54:10.773725 8559 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37409
I20260927 03:54:10.773763 8559 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:10.773819 8559 heartbeater.cc:511] Master 127.1.50.126:37409 requested a full tablet report, sending...
I20260927 03:54:10.774041 7316 ts_manager.cc:194] Re-registered known tserver with Master: a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:10.774503 7316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:41457
I20260927 03:54:10.778965 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.774835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.0.0.1:36484)
0927 03:54:10.774901 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.774909 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.774952 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.776020 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.776021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.776022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.776055 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.776056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.776853 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.776854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.777955 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.777958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.778784 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.778785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.778840 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.778842 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.778867 (+ 25us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.778904 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 03:54:10.779313 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.775784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.1.50.124:40027)
0927 03:54:10.775896 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.775897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.775905 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:10.775922 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.775923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.775941 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.776004 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.776110 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.776761 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.778113 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.778695 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.779114 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.779198 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.779215 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.779249 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260927 03:54:10.779341 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.771847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.1.50.126:37409)
0927 03:54:10.772354 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.772356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.772362 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:54:10.772380 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.772381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.772391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.773102 (+ 711us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.773140 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.773820 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.774808 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.775076 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.775129 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.776280 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.776281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.776281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.776288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.777963 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.778446 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.778447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.778447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.778501 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.778685 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.778686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.778686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.778798 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.778882 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.779185 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.779265 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":84,"threads_started":1}
I20260927 03:54:10.779484 8117 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 03:54:10.779604 8146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11715ff2510f4dbcb863ef34eb22f8a3. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.779686 8146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:54:10.780314 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.770939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.1.50.125:35635)
0927 03:54:10.771556 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.771558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.771680 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:54:10.771810 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.771812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.771822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.771900 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.772725 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.773455 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.774485 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.774753 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.774788 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.776959 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.776960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.776961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.776967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.778695 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.779512 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.779513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.779514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.779567 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.779772 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.779773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.779773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.779885 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.779946 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.780153 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.780232 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595}
I20260927 03:54:10.781838 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.770938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.1.50.124:40027)
0927 03:54:10.771580 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.771581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.774634 (+ 3053us) server_negotiation.cc:407] Connection header received
0927 03:54:10.774654 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.774655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.774666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.774714 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:10.775397 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.776123 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.777070 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.777331 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.777402 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.778904 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:10.778905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:10.778905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:10.778912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:10.780925 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.781206 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.781207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.781208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.781256 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:10.781416 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:10.781416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:10.781417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:10.781512 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:10.781600 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.781697 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.781761 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614}
I20260927 03:54:10.780763 7316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:10.782380 7316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:10.783032 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: NON_VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: true } } }
I20260927 03:54:10.783120 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.783681 7876 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: attempting to promote NON_VOTER 08371f7789b04ca1bf27663f4117b638 to VOTER
I20260927 03:54:10.783839 7876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:10.784134 7925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 03:54:10.784168 8117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 LEARNER]: Refusing update from remote peer 11715ff2510f4dbcb863ef34eb22f8a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 03:54:10.784322 7877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 03:54:10.784405 7875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 03:54:10.785048 7875 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08371f7789b04ca1bf27663f4117b638 (127.1.50.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.785135 8117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08371f7789b04ca1bf27663f4117b638 (127.1.50.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.785831 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.785898 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.785939 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.785982 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.786034 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.786079 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:10.786067 8146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.786147 8146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:54:10.786293 7925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08371f7789b04ca1bf27663f4117b638 (127.1.50.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.787061 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:10.787158 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:54:10.792420 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.788832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:46253 (local address 127.0.0.1:41270)
0927 03:54:10.788966 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.788977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.789017 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.789456 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.789457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.789458 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.789505 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.789507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790427 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.790428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.791288 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.791290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792176 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.792234 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.792238 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.792297 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.792340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260927 03:54:10.792434 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.789123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.1.50.65:46253)
0927 03:54:10.789276 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.789282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.789292 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:10.789333 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.789335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.789351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.789456 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.789557 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790361 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.791355 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792094 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792168 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.792247 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.792316 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.792353 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260927 03:54:10.792822 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.788835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42101 (local address 127.0.0.1:45240)
0927 03:54:10.788970 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.788978 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.789017 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.789634 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.789634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.789636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.789664 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.789665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790927 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.790928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.791760 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.791762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792617 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.792684 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.792690 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.792719 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.792757 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":20,"thread_start_us":91,"threads_started":1}
I20260927 03:54:10.793146 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.789130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.1.50.66:36331)
0927 03:54:10.789507 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.789510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.789519 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:10.789552 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.789553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.789566 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.789798 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.789848 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790919 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792158 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792814 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792879 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.792949 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.793029 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.793068 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":79,"threads_started":1}
I20260927 03:54:10.793264 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.789311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.1.50.67:42101)
0927 03:54:10.789440 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.789445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.789456 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:10.789530 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.789532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.789548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.789646 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.789728 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790845 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.791833 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792420 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.793029 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.793109 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:10.793134 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:54:10.793178 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260927 03:54:10.793210 8220 tablet_service.cc:1467] Processing CreateTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 (DEFAULT_TABLE table=test-table [id=36a68b3144b44e80af81fa56c1736860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:10.793545 8220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a89e5bd5cf48a09e0dce20c305f7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:10.793115 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.788797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:36331 (local address 127.0.0.1:47594)
0927 03:54:10.788967 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.788977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.789026 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.789669 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.789670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.789672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.789726 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.789728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.790973 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.790974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.792051 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.792053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.792895 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.792896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:10.792958 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:10.792962 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:10.793014 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:54:10.793052 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 03:54:10.794126 8493 tablet_service.cc:1467] Processing CreateTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 (DEFAULT_TABLE table=test-table [id=36a68b3144b44e80af81fa56c1736860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:10.794440 8493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a89e5bd5cf48a09e0dce20c305f7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:10.794418 8356 tablet_service.cc:1467] Processing CreateTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 (DEFAULT_TABLE table=test-table [id=36a68b3144b44e80af81fa56c1736860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:10.794730 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a89e5bd5cf48a09e0dce20c305f7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:10.796330 8587 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Bootstrap starting.
I20260927 03:54:10.796907 8587 tablet_bootstrap.cc:654] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:10.797161 8587 log.cc:824] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:10.797639 8589 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Bootstrap starting.
I20260927 03:54:10.797876 8587 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: No bootstrap required, opened a new log
I20260927 03:54:10.798000 8587 ts_tablet_manager.cc:1397] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:10.798270 8589 tablet_bootstrap.cc:654] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:10.798605 8589 log.cc:824] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:10.799010 8590 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Bootstrap starting.
I20260927 03:54:10.799365 8589 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: No bootstrap required, opened a new log
I20260927 03:54:10.799422 8589 ts_tablet_manager.cc:1397] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:10.799551 8590 tablet_bootstrap.cc:654] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:10.799823 8590 log.cc:824] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:10.800561 8590 tablet_bootstrap.cc:492] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: No bootstrap required, opened a new log
I20260927 03:54:10.800619 8590 ts_tablet_manager.cc:1397] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:10.801008 8587 raft_consensus.cc:348] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.801170 8587 raft_consensus.cc:374] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:10.801199 8587 raft_consensus.cc:729] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadee1d9cc3a4d1d9f2aacb3a04493a7, State: Initialized, Role: FOLLOWER
I20260927 03:54:10.801347 8587 consensus_queue.cc:260] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.801626 8587 ts_tablet_manager.cc:1428] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:54:10.801769 8286 heartbeater.cc:503] Master 127.1.50.126:37409 was elected leader, sending a full tablet report...
I20260927 03:54:10.801671 8589 raft_consensus.cc:348] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.801867 8589 raft_consensus.cc:374] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:10.801908 8589 raft_consensus.cc:729] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f32d5414c34a6490f550a012ec36c1, State: Initialized, Role: FOLLOWER
I20260927 03:54:10.802045 8589 consensus_queue.cc:260] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.802297 8589 ts_tablet_manager.cc:1428] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:10.802409 8559 heartbeater.cc:503] Master 127.1.50.126:37409 was elected leader, sending a full tablet report...
I20260927 03:54:10.803550 8590 raft_consensus.cc:348] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.803681 8590 raft_consensus.cc:374] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:10.803710 8590 raft_consensus.cc:729] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bb4f5c19da4e878b9d9ca8f83f7b67, State: Initialized, Role: FOLLOWER
I20260927 03:54:10.803821 8590 consensus_queue.cc:260] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.804046 8590 ts_tablet_manager.cc:1428] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:10.804109 8423 heartbeater.cc:503] Master 127.1.50.126:37409 was elected leader, sending a full tablet report...
W20260927 03:54:10.877449 8426 tablet.cc:2369] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:10.997225 8594 raft_consensus.cc:482] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:10.997354 8594 raft_consensus.cc:504] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:10.997745 8594 leader_election.cc:288] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331), cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
W20260927 03:54:10.998932 8289 tablet.cc:2369] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:11.001578 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.998043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:33553 (local address 127.1.50.66:36331)
0927 03:54:10.998100 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.998102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.998308 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:54:10.998357 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.998358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.998370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.998410 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.998529 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.999245 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.000737 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.001222 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.001297 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.001320 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.001465 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.001508 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:54:11.001585 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.997967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:36331 (local address 127.1.50.67:33553)
0927 03:54:10.998248 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.998258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.998303 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.998425 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.998426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.998428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.998475 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.998476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.999346 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.999347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:11.000653 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:11.000657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.001325 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.001326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:11.001400 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.001403 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:11.001449 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:54:11.001488 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":241}
I20260927 03:54:11.001616 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.998070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:53961 (local address 127.1.50.65:46253)
0927 03:54:10.998114 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:10.998116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:10.998133 (+ 17us) server_negotiation.cc:407] Connection header received
0927 03:54:10.998209 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:10.998209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:10.998220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:10.998284 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:10.998459 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.999246 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.000761 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.001266 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.001340 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.001349 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.001512 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.001557 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:11.001847 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:10.997956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:46253 (local address 127.1.50.67:53961)
0927 03:54:10.998086 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:10.998100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:10.998150 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:10.998303 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:10.998303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:10.998307 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:10.998385 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:10.998387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:10.999346 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:10.999347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:11.000672 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:11.000674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.001356 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.001357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:11.001446 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.001449 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:11.001491 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:54:11.001534 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":108}
I20260927 03:54:11.001936 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a89e5bd5cf48a09e0dce20c305f7e9" candidate_uuid: "a0f32d5414c34a6490f550a012ec36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" is_pre_election: true
I20260927 03:54:11.002149 8377 raft_consensus.cc:2454] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f32d5414c34a6490f550a012ec36c1 in term 0.
I20260927 03:54:11.002164 8240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a89e5bd5cf48a09e0dce20c305f7e9" candidate_uuid: "a0f32d5414c34a6490f550a012ec36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" is_pre_election: true
I20260927 03:54:11.002300 8240 raft_consensus.cc:2454] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f32d5414c34a6490f550a012ec36c1 in term 0.
I20260927 03:54:11.002374 8449 leader_election.cc:302] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0f32d5414c34a6490f550a012ec36c1, c7bb4f5c19da4e878b9d9ca8f83f7b67; no voters:
I20260927 03:54:11.002491 8594 raft_consensus.cc:2781] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:11.002554 8594 raft_consensus.cc:482] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:11.002573 8594 raft_consensus.cc:3037] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:11.003253 8594 raft_consensus.cc:504] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:11.003366 8594 leader_election.cc:288] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [CANDIDATE]: Term 1 election: Requested vote from peers c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331), cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253)
I20260927 03:54:11.003501 8240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a89e5bd5cf48a09e0dce20c305f7e9" candidate_uuid: "a0f32d5414c34a6490f550a012ec36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7"
I20260927 03:54:11.003585 8240 raft_consensus.cc:3037] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:11.004302 8240 raft_consensus.cc:2454] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f32d5414c34a6490f550a012ec36c1 in term 1.
I20260927 03:54:11.004390 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a89e5bd5cf48a09e0dce20c305f7e9" candidate_uuid: "a0f32d5414c34a6490f550a012ec36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
I20260927 03:54:11.004462 8447 leader_election.cc:302] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0f32d5414c34a6490f550a012ec36c1, cadee1d9cc3a4d1d9f2aacb3a04493a7; no voters:
I20260927 03:54:11.004463 8377 raft_consensus.cc:3037] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:11.004558 8594 raft_consensus.cc:2781] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:11.004690 8594 raft_consensus.cc:686] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 1 LEADER]: Becoming Leader. State: Replica: a0f32d5414c34a6490f550a012ec36c1, State: Running, Role: LEADER
I20260927 03:54:11.004923 8594 consensus_queue.cc:237] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } }
I20260927 03:54:11.005245 8377 raft_consensus.cc:2454] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f32d5414c34a6490f550a012ec36c1 in term 1.
I20260927 03:54:11.005685 7315 catalog_manager.cc:5495] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67). New cstate: current_term: 1 leader_uuid: "a0f32d5414c34a6490f550a012ec36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } health_report { overall_health: UNKNOWN } } }
W20260927 03:54:11.010447 8562 tablet.cc:2369] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:11.020833 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:11.018482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.1.50.67:42101)
0927 03:54:11.018533 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:11.018535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:11.018543 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:11.018581 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:11.018581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:11.018600 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:11.018656 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:11.018822 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.019289 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.020059 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.020263 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.020317 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.020417 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:11.020515 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:11.020570 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.020759 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.020776 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:54:11.022545 8377 raft_consensus.cc:1261] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Refusing update from remote peer a0f32d5414c34a6490f550a012ec36c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:11.022542 8240 raft_consensus.cc:1261] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Refusing update from remote peer a0f32d5414c34a6490f550a012ec36c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:11.022864 8594 consensus_queue.cc:1035] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:11.022997 8596 consensus_queue.cc:1035] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:11.026413 8600 mvcc.cc:205] Tried to move back new op lower bound from 7333811204185731072 to 7333811204116754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333811204185731072}]
I20260927 03:54:11.027092 8599 mvcc.cc:205] Tried to move back new op lower bound from 7333811204185731072 to 7333811204116754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333811204185731072 or (T in {7333811204185731072})}]
I20260927 03:54:11.027205 8601 mvcc.cc:205] Tried to move back new op lower bound from 7333811204185731072 to 7333811204116754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333811204185731072 or (T in {7333811204185731072})}]
I20260927 03:54:11.028432 7316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:11.028512 7316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:11.029719 7316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11715ff2510f4dbcb863ef34eb22f8a3: Sending DeleteTablet for 3 replicas of tablet 54a89e5bd5cf48a09e0dce20c305f7e9
I20260927 03:54:11.030001 8220 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:11 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:41270
I20260927 03:54:11.030089 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:11 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:47594
I20260927 03:54:11.030299 8610 tablet_replica.cc:330] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: stopping tablet replica
I20260927 03:54:11.030395 8610 raft_consensus.cc:2229] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:11.030498 8610 raft_consensus.cc:2258] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:11.030506 8609 tablet_replica.cc:330] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: stopping tablet replica
I20260927 03:54:11.030572 8493 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a89e5bd5cf48a09e0dce20c305f7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:11 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:45240
I20260927 03:54:11.030653 8609 raft_consensus.cc:2229] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:11.030798 8609 raft_consensus.cc:2258] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:11.030828 8611 tablet_replica.cc:330] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: stopping tablet replica
I20260927 03:54:11.030985 8611 raft_consensus.cc:2229] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:11.031167 8611 raft_consensus.cc:2258] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:11.031298 8609 ts_tablet_manager.cc:1905] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:11.031728 8611 ts_tablet_manager.cc:1905] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:11.032145 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7284
I20260927 03:54:11.033203 8609 ts_tablet_manager.cc:1918] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:11.033270 8609 log.cc:1196] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/54a89e5bd5cf48a09e0dce20c305f7e9
I20260927 03:54:11.033528 8609 ts_tablet_manager.cc:1939] T 54a89e5bd5cf48a09e0dce20c305f7e9 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Deleting consensus metadata
I20260927 03:54:11.033548 8611 ts_tablet_manager.cc:1918] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:11.033710 8611 log.cc:1196] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/54a89e5bd5cf48a09e0dce20c305f7e9
I20260927 03:54:11.033994 8611 ts_tablet_manager.cc:1939] T 54a89e5bd5cf48a09e0dce20c305f7e9 P a0f32d5414c34a6490f550a012ec36c1: Deleting consensus metadata
W20260927 03:54:11.037258 8175 connection.cc:535] server connection from 127.0.0.1:41270 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:54:11.037288 8446 connection.cc:535] server connection from 127.0.0.1:45240 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:54:11.038254 8610 ts_tablet_manager.cc:1905] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:11.039915 8610 ts_tablet_manager.cc:1918] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:11.039983 8610 log.cc:1196] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/54a89e5bd5cf48a09e0dce20c305f7e9
I20260927 03:54:11.040227 8610 ts_tablet_manager.cc:1939] T 54a89e5bd5cf48a09e0dce20c305f7e9 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting consensus metadata
W20260927 03:54:11.040572 8311 connection.cc:441] server connection from 127.0.0.1:47594 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47594 (request call id 1) could send its response
I20260927 03:54:11.046449 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:11.038835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.1.50.125:35635)
0927 03:54:11.038872 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:11.038874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:11.039274 (+ 400us) server_negotiation.cc:407] Connection header received
0927 03:54:11.039370 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:11.039371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:11.039381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:11.039408 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:11.040158 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.040841 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.041856 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.042183 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.042220 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.043683 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:11.043684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:11.043685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:11.043692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:11.045221 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.045582 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.045583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.045584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.045639 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.045827 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.045828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.045828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.045945 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:11.046013 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.046279 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.046356 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:11.047019 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:11.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.1.50.124:40027)
0927 03:54:11.038900 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:11.038901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:11.039251 (+ 350us) server_negotiation.cc:407] Connection header received
0927 03:54:11.039349 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:11.039350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:11.039363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:11.039390 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:11.040230 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.040984 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.042137 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.042450 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.042486 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.043754 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:11.043755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:11.043756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:11.043763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:11.045412 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.045848 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.045849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.045849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.045902 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.046127 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.046128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.046128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.046264 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:11.046333 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.046830 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.046929 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:54:11.048334 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:11.031683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.1.50.124:40027)
0927 03:54:11.031721 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:11.031723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:11.040733 (+ 9010us) server_negotiation.cc:407] Connection header received
0927 03:54:11.040812 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:11.040813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:11.040823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:11.040848 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:11.041522 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.042225 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.043591 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:11.043856 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:11.043894 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:11.045915 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:11.045916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:11.045917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:11.045923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:11.047369 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.047703 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.047704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.047704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.047754 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:11.047926 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:11.047927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:11.047927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:11.048020 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:11.048093 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:11.048193 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:54:11.048257 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:54:12.026945 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.026545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:43881)
0927 03:54:12.026804 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.026888 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
W20260927 03:54:12.027311 8423 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:37409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.027567 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.027256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:34661)
0927 03:54:12.027467 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.027526 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":47,"threads_started":1}
W20260927 03:54:12.027706 8286 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:37409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.027717 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.027478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:32999)
0927 03:54:12.027613 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.027677 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":46,"threads_started":1}
W20260927 03:54:12.027856 8559 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:37409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.530401 8623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11715ff2510f4dbcb863ef34eb22f8a3)
I20260927 03:54:12.530553 8623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:12.530786 8624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11715ff2510f4dbcb863ef34eb22f8a3)
I20260927 03:54:12.530854 8623 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027)
I20260927 03:54:12.530969 8624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:12.531318 8624 leader_election.cc:288] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125:35635)
I20260927 03:54:12.531364 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33534)
0927 03:54:12.531259 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.531319 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":62,"threads_started":1}
W20260927 03:54:12.531702 7902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.531716 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33540)
0927 03:54:12.531595 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.531669 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":42,"threads_started":1}
W20260927 03:54:12.531838 8094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.532271 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.532191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33562)
0927 03:54:12.532225 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.532247 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 03:54:12.532275 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.532188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33556)
0927 03:54:12.532224 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.532247 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 03:54:12.532359 8094 leader_election.cc:334] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
W20260927 03:54:12.532521 7902 leader_election.cc:334] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.534451 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:40027 (local address 127.0.0.1:36502)
0927 03:54:12.531165 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.531179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:12.531252 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:12.531655 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:12.531657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:12.531660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:12.531715 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:12.531717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.532646 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.532647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:12.533507 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:12.533509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.534158 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.534159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:12.534223 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:12.534229 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:54:12.534327 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:54:12.534377 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":50,"threads_started":1}
I20260927 03:54:12.534462 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.1.50.124:40027)
0927 03:54:12.531470 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:12.531473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:12.531484 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:12.531516 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:12.531517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:12.531529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:12.531595 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:12.531796 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.532557 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.533598 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.534088 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.534145 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:12.534156 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:12.534341 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 03:54:12.534389 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260927 03:54:12.534745 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.1.50.125:35635)
0927 03:54:12.531747 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:12.531749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:12.531756 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:12.531781 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:12.531781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:12.531792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:12.531847 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:12.532076 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.532828 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.533898 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.534401 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.534459 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:12.534472 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:12.534647 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:54:12.534691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":47,"threads_started":1}
I20260927 03:54:12.534768 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.531447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:35635 (local address 127.0.0.1:52974)
0927 03:54:12.531599 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.531607 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:12.531665 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:12.531903 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:12.531903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:12.531905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:12.531984 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:12.531986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.532914 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.532915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:12.533813 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:12.533814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:12.534510 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:12.534511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:12.534585 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:12.534590 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:12.534642 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:54:12.534703 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260927 03:54:12.534971 7925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08371f7789b04ca1bf27663f4117b638" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ee504b3d98541318bb7b10057405cf5" is_pre_election: true
I20260927 03:54:12.534955 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ee504b3d98541318bb7b10057405cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08371f7789b04ca1bf27663f4117b638" is_pre_election: true
I20260927 03:54:12.535085 8117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee504b3d98541318bb7b10057405cf5 in term 1.
I20260927 03:54:12.535084 7925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08371f7789b04ca1bf27663f4117b638 in term 1.
I20260927 03:54:12.535269 7900 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08371f7789b04ca1bf27663f4117b638, 0ee504b3d98541318bb7b10057405cf5; no voters: 11715ff2510f4dbcb863ef34eb22f8a3
I20260927 03:54:12.535279 8091 leader_election.cc:302] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08371f7789b04ca1bf27663f4117b638, 0ee504b3d98541318bb7b10057405cf5; no voters: 11715ff2510f4dbcb863ef34eb22f8a3
I20260927 03:54:12.535372 8623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:54:12.535415 8623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 11715ff2510f4dbcb863ef34eb22f8a3)
I20260927 03:54:12.535411 8624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:54:12.535439 8623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:54:12.535442 8624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 11715ff2510f4dbcb863ef34eb22f8a3)
I20260927 03:54:12.535465 8624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:54:12.536240 8623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:12.536301 8624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:12.536405 8623 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 election: Requested vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027)
I20260927 03:54:12.536427 8624 leader_election.cc:288] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 election: Requested vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 0ee504b3d98541318bb7b10057405cf5 (127.1.50.125:35635)
I20260927 03:54:12.536686 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.536516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33568)
0927 03:54:12.536610 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.536637 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 03:54:12.536654 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ee504b3d98541318bb7b10057405cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08371f7789b04ca1bf27663f4117b638"
I20260927 03:54:12.536633 7925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08371f7789b04ca1bf27663f4117b638" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ee504b3d98541318bb7b10057405cf5"
I20260927 03:54:12.536752 8117 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ee504b3d98541318bb7b10057405cf5 in current term 2: Already voted for candidate 08371f7789b04ca1bf27663f4117b638 in this term.
I20260927 03:54:12.536767 7925 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 08371f7789b04ca1bf27663f4117b638 in current term 2: Already voted for candidate 0ee504b3d98541318bb7b10057405cf5 in this term.
I20260927 03:54:12.537001 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.536507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33576)
0927 03:54:12.536716 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.536973 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 03:54:12.537082 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.536988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33586)
0927 03:54:12.537035 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.537060 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:54:12.537160 7902 leader_election.cc:334] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.537212 7902 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ee504b3d98541318bb7b10057405cf5; no voters: 08371f7789b04ca1bf27663f4117b638, 11715ff2510f4dbcb863ef34eb22f8a3
I20260927 03:54:12.537297 8623 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 03:54:12.537319 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:12.537236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33592)
0927 03:54:12.537277 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:12.537297 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:54:12.537380 8094 leader_election.cc:334] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:12.537431 8094 leader_election.cc:302] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08371f7789b04ca1bf27663f4117b638; no voters: 0ee504b3d98541318bb7b10057405cf5, 11715ff2510f4dbcb863ef34eb22f8a3
I20260927 03:54:12.537518 8624 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 03:54:13.028373 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:13.027731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:46243)
0927 03:54:13.028265 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:13.028313 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":94,"threads_started":1}
I20260927 03:54:13.028599 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:13.028158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:57309)
0927 03:54:13.028500 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:13.028534 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":89,"threads_started":1}
I20260927 03:54:13.028687 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:13.028266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:33161)
0927 03:54:13.028391 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:13.028435 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260927 03:54:14.029345 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.029077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:50911)
0927 03:54:14.029233 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.029274 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260927 03:54:14.029453 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.029201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:32819)
0927 03:54:14.029337 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.029384 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260927 03:54:14.029512 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.029267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:52377)
0927 03:54:14.029394 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.029424 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260927 03:54:14.117554 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:14.117673 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:14.117862 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027)
I20260927 03:54:14.118247 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ee504b3d98541318bb7b10057405cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08371f7789b04ca1bf27663f4117b638" is_pre_election: true
I20260927 03:54:14.118345 8117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee504b3d98541318bb7b10057405cf5 in term 2.
I20260927 03:54:14.118348 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.118127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33792)
0927 03:54:14.118246 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.118292 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260927 03:54:14.118530 7900 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08371f7789b04ca1bf27663f4117b638, 0ee504b3d98541318bb7b10057405cf5; no voters:
I20260927 03:54:14.118620 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 03:54:14.118669 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:14.118687 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:54:14.118826 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.118742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33804)
0927 03:54:14.118781 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.118803 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:54:14.118901 7902 leader_election.cc:334] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:14.119465 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:14.119603 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 election: Requested vote from peers 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409), 08371f7789b04ca1bf27663f4117b638 (127.1.50.124:40027)
I20260927 03:54:14.119765 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ee504b3d98541318bb7b10057405cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08371f7789b04ca1bf27663f4117b638"
I20260927 03:54:14.119832 8117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:54:14.119838 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.119715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33816)
0927 03:54:14.119790 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.119815 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 03:54:14.120119 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.120033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33824)
0927 03:54:14.120075 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.120100 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:54:14.120190 7902 leader_election.cc:334] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
I20260927 03:54:14.120599 8117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee504b3d98541318bb7b10057405cf5 in term 3.
I20260927 03:54:14.120929 7900 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08371f7789b04ca1bf27663f4117b638, 0ee504b3d98541318bb7b10057405cf5; no voters: 11715ff2510f4dbcb863ef34eb22f8a3
I20260927 03:54:14.121048 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:54:14.121204 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [term 3 LEADER]: Becoming Leader. State: Replica: 0ee504b3d98541318bb7b10057405cf5, State: Running, Role: LEADER
I20260927 03:54:14.121276 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } }
I20260927 03:54:14.121680 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee504b3d98541318bb7b10057405cf5. Latest consensus state: current_term: 3 leader_uuid: "0ee504b3d98541318bb7b10057405cf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:14.121809 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:14.122087 8644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:14.122536 8644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:14.122607 8644 catalog_manager.cc:1259] Loaded cluster ID: 4c6e834d1a7a4b3da70d4290f1eaf272
I20260927 03:54:14.122632 8644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:14.122813 8644 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:14.122887 8644 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Loaded TSK: 0
I20260927 03:54:14.123034 8644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:14.136724 7915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:14.138511 7915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:14.139708 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.139586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33834)
0927 03:54:14.139628 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.139663 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:54:14.139760 8117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [term 3 FOLLOWER]: Refusing update from remote peer 0ee504b3d98541318bb7b10057405cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 03:54:14.140024 8642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 03:54:14.140173 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.140036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.0.0.1:33842)
0927 03:54:14.140102 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.140141 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 03:54:14.140242 7902 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 -> Peer 11715ff2510f4dbcb863ef34eb22f8a3 (127.1.50.126:37409): Couldn't send request to peer 11715ff2510f4dbcb863ef34eb22f8a3. Status: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:54:14.140853 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee504b3d98541318bb7b10057405cf5. Latest consensus state: current_term: 3 leader_uuid: "0ee504b3d98541318bb7b10057405cf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:14.140954 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:54:14.141461 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0ee504b3d98541318bb7b10057405cf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:14.141564 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:14.141582 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0ee504b3d98541318bb7b10057405cf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "11715ff2510f4dbcb863ef34eb22f8a3" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37409 } } peers { permanent_uuid: "0ee504b3d98541318bb7b10057405cf5" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "08371f7789b04ca1bf27663f4117b638" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 40027 } attrs { promote: false } } }
I20260927 03:54:14.141659 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08371f7789b04ca1bf27663f4117b638 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:54:14.146986 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:46253 (local address 127.0.0.1:41286)
0927 03:54:14.143812 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.143824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:14.143892 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:14.144129 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:14.144130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:14.144131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:14.144197 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.144199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.144852 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.144853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.146132 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.146135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.146747 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.146748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.146813 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.146817 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:14.146862 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:54:14.146916 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 03:54:14.146986 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.1.50.65:46253)
0927 03:54:14.144023 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.144025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.144033 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:14.144060 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.144061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.144071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.144130 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:14.144265 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.144777 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.146224 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.146677 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.146739 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.146752 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.146871 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.146912 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":51,"threads_started":1}
I20260927 03:54:14.147446 8220 tablet_service.cc:1467] Processing CreateTablet for tablet 406aafdd988a49c5bda6db64f39d3092 (DEFAULT_TABLE table=test-table [id=fbba37ba894c432a8128ed9fbf327b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:14.147655 8220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aafdd988a49c5bda6db64f39d3092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:14.147716 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.1.50.67:42101)
0927 03:54:14.144352 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.144355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.144380 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:54:14.144437 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.144438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.144450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.144526 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:14.144659 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.145273 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.146403 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.147002 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.147340 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.147362 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.147533 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.147620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":80,"threads_started":1}
I20260927 03:54:14.147946 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42101 (local address 127.0.0.1:45246)
0927 03:54:14.144328 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.144337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:14.144387 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:14.144531 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:14.144532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:14.144534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:14.144599 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.144600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.145335 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.145336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.146303 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.146305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.147384 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.147385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.147440 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.147445 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:14.147498 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:54:14.147636 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":65,"threads_started":1}
I20260927 03:54:14.148064 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:36331 (local address 127.0.0.1:47608)
0927 03:54:14.144174 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.144185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:14.144254 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:14.145012 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:14.145013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:14.145015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:14.145084 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.145086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.145954 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.145955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.147147 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.147149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.147837 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.147838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.147916 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.147921 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:14.147945 (+ 24us) client_negotiation.cc:240] Negotiation successful
0927 03:54:14.147989 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":31,"thread_start_us":103,"threads_started":1}
I20260927 03:54:14.148214 8493 tablet_service.cc:1467] Processing CreateTablet for tablet 406aafdd988a49c5bda6db64f39d3092 (DEFAULT_TABLE table=test-table [id=fbba37ba894c432a8128ed9fbf327b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:14.148324 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.143985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.1.50.66:36331)
0927 03:54:14.144122 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.144125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.144860 (+ 735us) server_negotiation.cc:407] Connection header received
0927 03:54:14.144895 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.144896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.144908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.144990 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:14.145150 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.145868 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.147224 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.147728 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.148181 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.148199 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.148217 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.148261 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260927 03:54:14.148339 8493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aafdd988a49c5bda6db64f39d3092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:14.149412 8656 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Bootstrap starting.
I20260927 03:54:14.149852 8657 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Bootstrap starting.
I20260927 03:54:14.149947 8656 tablet_bootstrap.cc:654] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:14.150051 8356 tablet_service.cc:1467] Processing CreateTablet for tablet 406aafdd988a49c5bda6db64f39d3092 (DEFAULT_TABLE table=test-table [id=fbba37ba894c432a8128ed9fbf327b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:14.150198 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aafdd988a49c5bda6db64f39d3092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:14.150331 8657 tablet_bootstrap.cc:654] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:14.151042 8657 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: No bootstrap required, opened a new log
I20260927 03:54:14.151114 8657 ts_tablet_manager.cc:1397] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:14.151249 8657 raft_consensus.cc:348] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.151332 8657 raft_consensus.cc:374] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:14.151350 8657 raft_consensus.cc:729] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f32d5414c34a6490f550a012ec36c1, State: Initialized, Role: FOLLOWER
I20260927 03:54:14.151420 8657 consensus_queue.cc:260] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.151628 8657 ts_tablet_manager.cc:1428] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:14.151966 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.151847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:60163)
0927 03:54:14.151903 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.151935 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:54:14.152247 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.152159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:48965)
0927 03:54:14.152203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.152225 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:54:14.152350 8561 heartbeater.cc:503] Master 127.1.50.125:35635 was elected leader, sending a full tablet report...
I20260927 03:54:14.152522 8661 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Bootstrap starting.
I20260927 03:54:14.152977 8661 tablet_bootstrap.cc:654] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:14.153609 8661 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: No bootstrap required, opened a new log
I20260927 03:54:14.153602 8656 tablet_bootstrap.cc:492] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7: No bootstrap required, opened a new log
I20260927 03:54:14.153663 8661 ts_tablet_manager.cc:1397] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:14.153666 8656 ts_tablet_manager.cc:1397] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:54:14.153812 8661 raft_consensus.cc:348] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.153832 8656 raft_consensus.cc:348] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.153891 8661 raft_consensus.cc:374] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:14.153896 8656 raft_consensus.cc:374] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:14.153911 8661 raft_consensus.cc:729] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bb4f5c19da4e878b9d9ca8f83f7b67, State: Initialized, Role: FOLLOWER
I20260927 03:54:14.153915 8656 raft_consensus.cc:729] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadee1d9cc3a4d1d9f2aacb3a04493a7, State: Initialized, Role: FOLLOWER
I20260927 03:54:14.153968 8656 consensus_queue.cc:260] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.154240 8656 ts_tablet_manager.cc:1428] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:14.153991 8661 consensus_queue.cc:260] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.154415 8661 ts_tablet_manager.cc:1428] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:14.154600 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.154421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:40407)
0927 03:54:14.154524 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.154559 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 03:54:14.154760 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.154613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:58587)
0927 03:54:14.154681 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.154725 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:54:14.154946 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.154866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:60205)
0927 03:54:14.154903 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.154924 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:54:14.155426 8288 heartbeater.cc:503] Master 127.1.50.125:35635 was elected leader, sending a full tablet report...
I20260927 03:54:14.155504 8425 heartbeater.cc:503] Master 127.1.50.125:35635 was elected leader, sending a full tablet report...
I20260927 03:54:14.155594 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.155490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:42465)
0927 03:54:14.155541 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.155567 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:54:14.157428 8663 raft_consensus.cc:482] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:14.157498 8663 raft_consensus.cc:504] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.157866 8663 leader_election.cc:288] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331), a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:14.160966 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.157983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:36331 (local address 127.1.50.65:47805)
0927 03:54:14.158088 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.158102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:14.158179 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:14.158407 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:14.158408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:14.158412 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:14.158531 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.158534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.159219 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.159220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.160187 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.160190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.160717 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.160795 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.160800 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:14.160844 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:54:14.160898 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 03:54:14.160964 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.158064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:47805 (local address 127.1.50.66:36331)
0927 03:54:14.158129 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.158131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.158166 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:54:14.158241 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.158242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.158331 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.158393 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:14.158607 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.159147 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160268 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.160648 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160707 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.160721 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.160855 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.160898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:54:14.161053 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.158235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35249 (local address 127.1.50.67:42101)
0927 03:54:14.158295 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.158297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.158305 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:14.158398 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.158400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.158414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.158472 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:14.158702 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.159234 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160325 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.160700 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160761 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.160775 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.160938 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.160978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:54:14.161068 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.158003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42101 (local address 127.1.50.65:35249)
0927 03:54:14.158236 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.158247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:14.158312 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:14.158493 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:14.158494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:14.158496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:14.158582 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.158585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.159305 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.159306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.160242 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:14.160245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.160795 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.160796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:14.160875 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.160881 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:54:14.160927 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:54:14.160983 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260927 03:54:14.161382 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aafdd988a49c5bda6db64f39d3092" candidate_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" is_pre_election: true
I20260927 03:54:14.161473 8377 raft_consensus.cc:2454] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cadee1d9cc3a4d1d9f2aacb3a04493a7 in term 0.
I20260927 03:54:14.161468 8513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aafdd988a49c5bda6db64f39d3092" candidate_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f32d5414c34a6490f550a012ec36c1" is_pre_election: true
I20260927 03:54:14.161629 8513 raft_consensus.cc:2454] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cadee1d9cc3a4d1d9f2aacb3a04493a7 in term 0.
I20260927 03:54:14.161640 8176 leader_election.cc:302] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7bb4f5c19da4e878b9d9ca8f83f7b67, cadee1d9cc3a4d1d9f2aacb3a04493a7; no voters:
I20260927 03:54:14.161796 8663 raft_consensus.cc:2781] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:14.161834 8663 raft_consensus.cc:482] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:14.161850 8663 raft_consensus.cc:3037] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:14.162727 8663 raft_consensus.cc:504] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.162876 8663 leader_election.cc:288] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [CANDIDATE]: Term 1 election: Requested vote from peers c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331), a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101)
I20260927 03:54:14.163074 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aafdd988a49c5bda6db64f39d3092" candidate_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67"
I20260927 03:54:14.163141 8377 raft_consensus.cc:3037] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:14.163110 8513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aafdd988a49c5bda6db64f39d3092" candidate_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f32d5414c34a6490f550a012ec36c1"
I20260927 03:54:14.163194 8513 raft_consensus.cc:3037] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:14.163800 8377 raft_consensus.cc:2454] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cadee1d9cc3a4d1d9f2aacb3a04493a7 in term 1.
I20260927 03:54:14.163960 8176 leader_election.cc:302] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7bb4f5c19da4e878b9d9ca8f83f7b67, cadee1d9cc3a4d1d9f2aacb3a04493a7; no voters:
I20260927 03:54:14.164028 8513 raft_consensus.cc:2454] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cadee1d9cc3a4d1d9f2aacb3a04493a7 in term 1.
I20260927 03:54:14.164062 8663 raft_consensus.cc:2781] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:14.164215 8663 raft_consensus.cc:686] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [term 1 LEADER]: Becoming Leader. State: Replica: cadee1d9cc3a4d1d9f2aacb3a04493a7, State: Running, Role: LEADER
I20260927 03:54:14.164362 8663 consensus_queue.cc:237] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } }
I20260927 03:54:14.164629 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.164453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:49711)
0927 03:54:14.164547 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.164592 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 03:54:14.165241 7915 catalog_manager.cc:5495] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadee1d9cc3a4d1d9f2aacb3a04493a7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:14.175863 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.173416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.1.50.65:46253)
0927 03:54:14.173493 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:14.173495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:14.173506 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:14.173580 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:14.173581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:14.173603 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:14.173667 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:14.173797 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.174372 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.175126 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:14.175319 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:14.175374 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:14.175515 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:14.175634 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:14.175700 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:14.175738 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:54:14.175763 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:54:14.177472 8513 raft_consensus.cc:1261] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Refusing update from remote peer cadee1d9cc3a4d1d9f2aacb3a04493a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:14.177758 8377 raft_consensus.cc:1261] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Refusing update from remote peer cadee1d9cc3a4d1d9f2aacb3a04493a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:14.177933 8666 consensus_queue.cc:1035] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f32d5414c34a6490f550a012ec36c1" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:14.178117 8663 consensus_queue.cc:1035] T 406aafdd988a49c5bda6db64f39d3092 P cadee1d9cc3a4d1d9f2aacb3a04493a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bb4f5c19da4e878b9d9ca8f83f7b67" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:14.179260 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.179007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:51695)
0927 03:54:14.179183 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.179220 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 03:54:14.179412 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.179125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:53289)
0927 03:54:14.179349 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.179378 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260927 03:54:14.180054 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.179927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:40825)
0927 03:54:14.179977 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.180011 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:54:14.181000 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.180886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.65:33833)
0927 03:54:14.180945 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.180970 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:54:14.181052 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.180945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.66:43605)
0927 03:54:14.181005 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.181027 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 03:54:14.181461 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:14.181300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37409 (local address 127.1.50.67:41365)
0927 03:54:14.181403 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.181432 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 03:54:14.184347 7913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:14.184446 7913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:14.185941 7913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ee504b3d98541318bb7b10057405cf5: Sending DeleteTablet for 3 replicas of tablet 406aafdd988a49c5bda6db64f39d3092
I20260927 03:54:14.186421 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aafdd988a49c5bda6db64f39d3092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:14 UTC) from {username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:47608
I20260927 03:54:14.186556 8493 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aafdd988a49c5bda6db64f39d3092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:14 UTC) from {username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:45246
I20260927 03:54:14.186641 8680 tablet_replica.cc:330] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: stopping tablet replica
I20260927 03:54:14.186694 8680 raft_consensus.cc:2229] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:14.186944 8220 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aafdd988a49c5bda6db64f39d3092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:14 UTC) from {username='oryx', principal='oryx/127.1.50.125@KRBTEST.COM'} at 127.0.0.1:41286
I20260927 03:54:14.186981 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8062
I20260927 03:54:14.187712 8681 tablet_replica.cc:330] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: stopping tablet replica
I20260927 03:54:14.187780 8681 raft_consensus.cc:2229] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:14.187844 8681 raft_consensus.cc:2258] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:14.188967 8680 raft_consensus.cc:2258] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:14.189348 8680 ts_tablet_manager.cc:1905] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:14.190402 8681 ts_tablet_manager.cc:1905] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:14.190963 8680 ts_tablet_manager.cc:1918] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:14.191023 8680 log.cc:1196] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/406aafdd988a49c5bda6db64f39d3092
I20260927 03:54:14.191236 8680 ts_tablet_manager.cc:1939] T 406aafdd988a49c5bda6db64f39d3092 P c7bb4f5c19da4e878b9d9ca8f83f7b67: Deleting consensus metadata
I20260927 03:54:14.191672 7902 catalog_manager.cc:4841] TS c7bb4f5c19da4e878b9d9ca8f83f7b67 (127.1.50.66:36331): tablet 406aafdd988a49c5bda6db64f39d3092 (table test-table [id=fbba37ba894c432a8128ed9fbf327b2a]) successfully deleted
I20260927 03:54:14.192210 8681 ts_tablet_manager.cc:1918] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:14.192270 8681 log.cc:1196] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/406aafdd988a49c5bda6db64f39d3092
W20260927 03:54:14.192327 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:14.192152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:46253 (local address 127.0.0.1:41310)
0927 03:54:14.192221 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:14.192254 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:46253: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 03:54:14.192423 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8294
I20260927 03:54:14.192507 8681 ts_tablet_manager.cc:1939] T 406aafdd988a49c5bda6db64f39d3092 P a0f32d5414c34a6490f550a012ec36c1: Deleting consensus metadata
W20260927 03:54:14.192595 7900 catalog_manager.cc:4584] TS cadee1d9cc3a4d1d9f2aacb3a04493a7 (127.1.50.65:46253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 406aafdd988a49c5bda6db64f39d3092: Network error: Client connection negotiation failed: client connection to 127.1.50.65:46253: connect: Connection reset by peer (error 104)
I20260927 03:54:14.192802 7901 catalog_manager.cc:4841] TS a0f32d5414c34a6490f550a012ec36c1 (127.1.50.67:42101): tablet 406aafdd988a49c5bda6db64f39d3092 (table test-table [id=fbba37ba894c432a8128ed9fbf327b2a]) successfully deleted
I20260927 03:54:14.198228 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8430
I20260927 03:54:14.204886 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7780
I20260927 03:54:14.210716 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 7969
2026-09-27T03:54:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6122 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:14 dist-test-slave-96fg krb5kdc[8684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:14 dist-test-slave-96fg krb5kdc[8684](info): set up 2 sockets
Sep 27 03:54:14 dist-test-slave-96fg krb5kdc[8684](info): commencing operation
krb5kdc: starting...
W20260927 03:54:15.170318 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab.
Entry for principal oryx/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab.
I20260927 03:54:15.290836 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38053
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab
--principal=oryx/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:15.371630 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:15.371860 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:15.371881 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:15.373041 8697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:15.373081 8697 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:54:15.373093 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:15.373104 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:15.373116 8697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:15.373126 8697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:15.374651 8697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:38053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.126.keytab
--principal=oryx/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:15.374897 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:15.375106 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:15.379932 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.380043 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:15.380074 8697 server_base.cc:1034] running on GCE node
W20260927 03:54:15.379973 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:15.380544 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:15.380860 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:15.382012 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790481255381995 us; error 36 us; skew 500 ppm
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:15.384506 8697 init.cc:377] Logged in from keytab as oryx/127.1.50.126@KRBTEST.COM (short username oryx)
I20260927 03:54:15.384948 8697 webserver.cc:460] Webserver started at http://127.1.50.126:45245/ using document root <none> and password file <none>
I20260927 03:54:15.385159 8697 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:15.385201 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:15.385337 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:15.386447 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "9fb02728e397478293b7faf3292787b2"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "ade60fd5e1edaf014220de4f31cac9ef"
server_key_iv: "1d317f72f0a4660ad9ba352a70a24362"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.386762 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "9fb02728e397478293b7faf3292787b2"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "ade60fd5e1edaf014220de4f31cac9ef"
server_key_iv: "1d317f72f0a4660ad9ba352a70a24362"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.387986 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:15.388706 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.388890 8697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:15.388972 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9fb02728e397478293b7faf3292787b2"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "ade60fd5e1edaf014220de4f31cac9ef"
server_key_iv: "1d317f72f0a4660ad9ba352a70a24362"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.389030 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:15.405472 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:15.407652 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:15.407799 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:15.412771 8697 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:38053
I20260927 03:54:15.412856 8774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:38053 every 8 connection(s)
I20260927 03:54:15.413187 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:15.413813 8775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:15.415530 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8697
I20260927 03:54:15.415621 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:15.415799 1225 external_mini_cluster.cc:1428] Setting key 87cc25ffcbc7852b680af4651be0e3c5
I20260927 03:54:15.416378 8775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Bootstrap starting.
I20260927 03:54:15.416992 8775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:15.417265 8775 log.cc:824] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:15.417888 8775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: No bootstrap required, opened a new log
I20260927 03:54:15.419664 8775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } }
I20260927 03:54:15.419811 8775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:15.419845 8775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb02728e397478293b7faf3292787b2, State: Initialized, Role: FOLLOWER
I20260927 03:54:15.419971 8775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } }
I20260927 03:54:15.420044 8775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:15.420075 8775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:15.420117 8775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:15.420622 8775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } }
I20260927 03:54:15.420717 8775 leader_election.cc:302] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb02728e397478293b7faf3292787b2; no voters:
I20260927 03:54:15.420863 8775 leader_election.cc:288] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:15.420900 8780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:15.421010 8780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9fb02728e397478293b7faf3292787b2, State: Running, Role: LEADER
I20260927 03:54:15.421088 8780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } }
I20260927 03:54:15.421118 8775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:15.421419 8780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb02728e397478293b7faf3292787b2. Latest consensus state: current_term: 1 leader_uuid: "9fb02728e397478293b7faf3292787b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } } }
I20260927 03:54:15.421432 8782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fb02728e397478293b7faf3292787b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb02728e397478293b7faf3292787b2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38053 } } }
I20260927 03:54:15.421492 8780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:15.421535 8782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:15.421720 8785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:15.422392 8785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:15.424024 8785 catalog_manager.cc:1347] Generated new cluster ID: 3f0ab5c4a98a4e29a911ee05ff9cc9a8
I20260927 03:54:15.424067 8785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:15.426342 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.416356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.1.50.126:38053)
0927 03:54:15.416530 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.416540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.416570 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:54:15.417062 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.417072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.417142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.417307 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.417869 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.418809 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.419520 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.419992 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.420093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.422821 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.422827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.422827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.422839 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.424747 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.425156 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.425159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.425160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.425259 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.425450 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.425451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.425451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.425733 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.425865 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.425982 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.426134 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260927 03:54:15.432848 8785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:15.433377 8785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:15.435886 8785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Generated new TSK 0
I20260927 03:54:15.436057 8785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
I20260927 03:54:15.465631 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38053
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:15.558508 8801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:15.558753 8801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:15.558781 8801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:15.560076 8801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:15.560128 8801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:15.560159 8801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:54:15.560194 8801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:15.561990 8801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:15.562280 8801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:15.562512 8801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:15.568440 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.568427 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.568502 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:15.568427 8801 server_base.cc:1034] running on GCE node
I20260927 03:54:15.568890 8801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:15.569159 8801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:15.570307 8801 hybrid_clock.cc:648] HybridClock initialized: now 1790481255570285 us; error 33 us; skew 500 ppm
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:15.572824 8801 init.cc:377] Logged in from keytab as oryx/127.1.50.65@KRBTEST.COM (short username oryx)
I20260927 03:54:15.573283 8801 webserver.cc:460] Webserver started at http://127.1.50.65:45105/ using document root <none> and password file <none>
I20260927 03:54:15.573518 8801 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:15.573567 8801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:15.573697 8801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:15.574839 8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "6e15455fdf79488ead3de5b4424777ae"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "b23b4740287adaba9264ef1ff0195647"
server_key_iv: "00150402963e4539f7411be2261343ba"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.575194 8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "6e15455fdf79488ead3de5b4424777ae"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "b23b4740287adaba9264ef1ff0195647"
server_key_iv: "00150402963e4539f7411be2261343ba"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.576637 8801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:54:15.577482 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.577659 8801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:15.577736 8801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "6e15455fdf79488ead3de5b4424777ae"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "b23b4740287adaba9264ef1ff0195647"
server_key_iv: "00150402963e4539f7411be2261343ba"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.577819 8801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:15.603801 8801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:15.605712 8801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:15.605840 8801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:15.606460 8801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:15.606498 8801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.606520 8801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:15.606531 8801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.612952 8801 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:34687
I20260927 03:54:15.613019 8931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:34687 every 8 connection(s)
I20260927 03:54:15.613327 8801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:15.616561 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.606936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44295 (local address 127.1.50.126:38053)
0927 03:54:15.607020 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.607022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.607236 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:54:15.607780 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.607782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.607794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.607850 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.608826 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.609618 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.611220 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.611428 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.611494 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.614185 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.614187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.614187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.614194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.615482 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.615834 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.615835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.615835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.615893 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.616074 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.616075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.616075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.616190 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.616237 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.616431 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.616499 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:54:15.616717 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.606908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.65:44295)
0927 03:54:15.607082 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.607122 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.607674 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.608008 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.608014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.608402 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.608710 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.608717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.609722 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.609723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.611093 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.611096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.611515 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.611516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.611773 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.612382 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.612392 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.614076 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.615555 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.615560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.615562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.615756 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.615948 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.615949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.615949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.616034 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.616232 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.616235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.616346 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.616419 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.616536 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260927 03:54:15.617180 8932 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38053
I20260927 03:54:15.617256 8932 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:15.617420 8932 heartbeater.cc:511] Master 127.1.50.126:38053 requested a full tablet report, sending...
I20260927 03:54:15.617914 8729 ts_manager.cc:194] Registered new tserver with Master: 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687)
I20260927 03:54:15.618700 8729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:44295
I20260927 03:54:15.621224 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8801
I20260927 03:54:15.621302 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:15.621490 1225 external_mini_cluster.cc:1428] Setting key 98116d6a0250f090b84ec535da337c6d
I20260927 03:54:15.623703 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.618160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:45827 (local address 127.1.50.126:38053)
0927 03:54:15.618193 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.618194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.618294 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:54:15.618335 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.618336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.618344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.618407 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.618721 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.619348 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.620049 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.620270 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.620333 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.621155 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.621157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.621157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.621164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.622660 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.623035 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.623036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.623037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.623099 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.623254 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.623254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.623255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.623375 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.623444 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.623575 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.623642 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:54:15.623822 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.618111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.65:45827)
0927 03:54:15.618231 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.618240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.618290 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.618409 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.618410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.618584 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.618661 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.618663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.619442 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.619443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.619958 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.619960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.620342 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.620343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.620388 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.620764 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.620768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.621058 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.622760 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.622762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.622763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.622952 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.623148 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.623149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.623149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.623199 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.623439 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.623439 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.623487 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.623553 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.623729 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
I20260927 03:54:15.645407 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38053
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:15.724309 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:15.724520 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:15.724545 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:15.725675 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:15.725715 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:15.725734 8941 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:54:15.725750 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:15.727257 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:15.727465 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:15.727648 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:15.732474 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.732463 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.732477 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:15.732772 8941 server_base.cc:1034] running on GCE node
I20260927 03:54:15.732981 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:15.733203 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:15.734359 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790481255734340 us; error 27 us; skew 500 ppm
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:15.736855 8941 init.cc:377] Logged in from keytab as oryx/127.1.50.66@KRBTEST.COM (short username oryx)
I20260927 03:54:15.737295 8941 webserver.cc:460] Webserver started at http://127.1.50.66:42715/ using document root <none> and password file <none>
I20260927 03:54:15.737507 8941 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:15.737555 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:15.737684 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:15.738780 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "4514a7b408bb4389bf680047b5fd739f"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "16fcbaacb7231601e9c1fc6cd05bad4c"
server_key_iv: "9ad5942d583cae25f5a717ebe17f94d4"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.739132 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "4514a7b408bb4389bf680047b5fd739f"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "16fcbaacb7231601e9c1fc6cd05bad4c"
server_key_iv: "9ad5942d583cae25f5a717ebe17f94d4"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.740494 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:15.741214 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.741449 8941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:15.741539 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "4514a7b408bb4389bf680047b5fd739f"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "16fcbaacb7231601e9c1fc6cd05bad4c"
server_key_iv: "9ad5942d583cae25f5a717ebe17f94d4"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.741600 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:15.749357 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:15.751945 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:15.752097 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:15.752599 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:15.752645 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.752678 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:15.752717 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.759138 8941 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:32805
I20260927 03:54:15.759281 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:32805 every 8 connection(s)
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:15.759526 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:15.760087 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 8941
I20260927 03:54:15.760185 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:15.760386 1225 external_mini_cluster.cc:1428] Setting key 3cd690869d093c2bc3ebd646fa718766
I20260927 03:54:15.762612 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.753130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:37241 (local address 127.1.50.126:38053)
0927 03:54:15.753232 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.753235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.753451 (+ 216us) server_negotiation.cc:407] Connection header received
0927 03:54:15.753960 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.753961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.753973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.754075 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.754861 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.755742 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.757247 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.757534 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.757597 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.759981 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.759983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.759983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.759991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.761285 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.761630 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.761638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.761639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.761693 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.761942 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.761943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.761944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.762103 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.762180 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.762430 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.762519 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 03:54:15.762681 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.753060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.66:37241)
0927 03:54:15.753316 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.753360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.753863 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.754137 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.754141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.754457 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.754723 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.754728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.755856 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.755858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.757159 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.757162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.757604 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.757605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.757730 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.758254 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.758271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.759879 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.761358 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.761364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.761367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.761568 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.761745 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.761746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.761746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.761852 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.762172 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.762178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.762310 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.762401 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.762521 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260927 03:54:15.763058 9072 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38053
I20260927 03:54:15.763139 9072 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:15.763301 9072 heartbeater.cc:511] Master 127.1.50.126:38053 requested a full tablet report, sending...
I20260927 03:54:15.763659 8728 ts_manager.cc:194] Registered new tserver with Master: 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805)
I20260927 03:54:15.764111 8728 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:37241
WARNING: no policy specified for oryx/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.67@KRBTEST.COM" created.
I20260927 03:54:15.771674 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.764109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.66:53953)
0927 03:54:15.766115 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.766126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.766188 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.766324 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.766325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.766491 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.766540 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.766541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.767448 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.767449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.767985 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.767986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.768476 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.768477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.768535 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.768992 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.768997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.769269 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.770535 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.770536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.770537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.770694 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.771010 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.771011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.771012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.771078 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.771357 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.771358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.771405 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.771459 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.771566 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":71,"threads_started":1}
I20260927 03:54:15.771674 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.764059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:53953 (local address 127.1.50.126:38053)
0927 03:54:15.764102 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.764103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.766191 (+ 2088us) server_negotiation.cc:407] Connection header received
0927 03:54:15.766247 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.766248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.766262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.766320 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.766601 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.767344 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.768067 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.768390 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.768464 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.769345 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.769346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.769346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.769353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.770470 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.770775 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.770776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.770776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.770845 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.771143 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.771145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.771145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.771296 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.771355 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.771491 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.771567 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
I20260927 03:54:15.786752 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:38053
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:15.866423 9081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:15.866647 9081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:15.866680 9081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:15.867870 9081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:15.867911 9081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:15.867940 9081 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:54:15.867965 9081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:15.869504 9081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.50.126:38053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:15.869746 9081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:15.869951 9081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:15.874851 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.874854 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:15.874876 9089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:15.875458 9081 server_base.cc:1034] running on GCE node
I20260927 03:54:15.875666 9081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:15.875880 9081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:15.877009 9081 hybrid_clock.cc:648] HybridClock initialized: now 1790481255876993 us; error 31 us; skew 500 ppm
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:15.879508 9081 init.cc:377] Logged in from keytab as oryx/127.1.50.67@KRBTEST.COM (short username oryx)
I20260927 03:54:15.879963 9081 webserver.cc:460] Webserver started at http://127.1.50.67:36523/ using document root <none> and password file <none>
I20260927 03:54:15.880164 9081 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:15.880213 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:15.880334 9081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:15.881441 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "3477eae3be45442bb2361fab45fd2418"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "1d019f147653073bc4d4ee6173daf7e4"
server_key_iv: "687641860bca7f1e509ae78d4cdfc1d6"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.881773 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "3477eae3be45442bb2361fab45fd2418"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "1d019f147653073bc4d4ee6173daf7e4"
server_key_iv: "687641860bca7f1e509ae78d4cdfc1d6"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.883071 9081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:15.883734 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.883893 9081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:15.883973 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "3477eae3be45442bb2361fab45fd2418"
format_stamp: "Formatted at 2026-09-27 03:54:15 on dist-test-slave-96fg"
server_key: "1d019f147653073bc4d4ee6173daf7e4"
server_key_iv: "687641860bca7f1e509ae78d4cdfc1d6"
server_key_version: "encryptionkey@0"
I20260927 03:54:15.884032 9081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:15.902652 9081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:15.904529 9081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:15.904667 9081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:15.905216 9081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:15.905263 9081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:15.905292 9081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:15.905319 9081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:15.912106 9081 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:41021
I20260927 03:54:15.912201 9211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:41021 every 8 connection(s)
I20260927 03:54:15.912505 9081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:15.914991 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.905679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:39233 (local address 127.1.50.126:38053)
0927 03:54:15.905744 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.905746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.905938 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:54:15.906701 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.906702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.906715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.906774 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.907618 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.908120 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.909168 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.909367 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.909427 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.912271 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.912273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.912273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.912281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.913666 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.914058 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.914059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.914059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.914130 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.914398 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.914399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.914399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.914529 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.914595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.914853 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.914925 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:54:15.915138 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.905652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.67:39233)
0927 03:54:15.905805 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.905857 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.906566 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.906858 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.906863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.907223 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.907516 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.907522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.908204 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.908206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.909082 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.909085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.909474 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.909476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.909586 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.910109 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.910117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.912165 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.913739 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.913744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.913746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.913954 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.914182 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.914182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.914182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.914278 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.914594 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.914599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.914725 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.914810 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.914932 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260927 03:54:15.915652 9212 heartbeater.cc:347] Connected to a master server at 127.1.50.126:38053
I20260927 03:54:15.915741 9212 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:15.915915 9212 heartbeater.cc:511] Master 127.1.50.126:38053 requested a full tablet report, sending...
I20260927 03:54:15.916287 8728 ts_manager.cc:194] Registered new tserver with Master: 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:15.916782 8728 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:39233
I20260927 03:54:15.921517 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9081
I20260927 03:54:15.921643 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:15.921855 1225 external_mini_cluster.cc:1428] Setting key 372bb53e5c792d11eefec44b59f0ddce
I20260927 03:54:15.922351 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.916654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:44537 (local address 127.1.50.126:38053)
0927 03:54:15.916706 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.916707 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.916770 (+ 63us) server_negotiation.cc:407] Connection header received
0927 03:54:15.916873 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.916874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.916884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.916941 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.917348 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.917897 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.918707 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.918906 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.918974 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.919815 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.919816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.919817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.919823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.921025 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.921378 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.921379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.921379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.921430 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.921718 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.921727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.921728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.921893 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.921977 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.922179 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.922266 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:54:15.922505 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.916589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.1.50.67:44537)
0927 03:54:15.916712 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.916723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.916784 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.916957 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.916958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.917155 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:15.917274 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.917277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.917990 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.917991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.918576 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.918578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.918973 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.918974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.919025 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.919438 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:15.919442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:15.919731 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:15.921103 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.921104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.921104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.921263 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.921489 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:15.921490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:15.921490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:15.921554 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:15.921968 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:15.921969 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:15.922038 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.922150 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.922269 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260927 03:54:15.923004 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:54:15.943259 8778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:15.937977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.1.50.126:38053)
0927 03:54:15.938044 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.938046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.938056 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:15.938148 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.938149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.938157 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.938204 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.938754 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.939240 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.939947 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.940122 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.940166 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.943188 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50480: BlockingRecv error: recv got EOF from 127.0.0.1:50480 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:54:15 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:15.950724 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.944098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.1.50.126:38053)
0927 03:54:15.944153 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.944155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.944188 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:54:15.944258 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.944259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.944268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.944336 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:15.944766 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.945258 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.945994 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.946219 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.946287 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.948323 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:15.948325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:15.948325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:15.948332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:15.949561 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.949968 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.949969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.949969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.950075 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:15.950320 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:15.950320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:15.950320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:15.950439 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:15.950515 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.950598 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.950658 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:54:15.952493 8729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:15.954002 8729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:15.960963 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.1.50.67:41021)
0927 03:54:15.956967 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.956970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.956986 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:54:15.957019 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.957021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.957034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.957187 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:15.957529 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958779 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.959892 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960599 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960768 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.960829 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.960867 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260927 03:54:15.960951 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:41021 (local address 127.0.0.1:35904)
0927 03:54:15.956811 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.956820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.956863 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.957365 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.957366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.957368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:15.957419 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.957421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958910 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.958911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.959797 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.959800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960702 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.960758 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960761 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.960804 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.960836 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":63,"threads_started":1}
I20260927 03:54:15.961130 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.1.50.66:32805)
0927 03:54:15.957188 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.957191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.957200 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:15.957233 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.957235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.957248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.957321 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:15.957506 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958813 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960020 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960513 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960576 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.961003 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.961040 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":87,"threads_started":1}
I20260927 03:54:15.960951 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:32805 (local address 127.0.0.1:38468)
0927 03:54:15.956561 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.956602 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.956662 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.957390 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.957391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.957393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:15.957436 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.957437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958892 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.958893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.959949 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.959951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960589 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.960690 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960706 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.960773 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.960830 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":38,"thread_start_us":57,"threads_started":1}
I20260927 03:54:15.961364 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.0.0.1:55974)
0927 03:54:15.956689 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:15.956700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:15.956749 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:15.957132 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:15.957136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:15.957139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:15.957219 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.957221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958480 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.958481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.959756 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:15.959758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960778 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:15.960840 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960844 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:15.960885 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:54:15.960926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":4,"thread_start_us":76,"threads_started":1}
I20260927 03:54:15.961373 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:15.956515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.1.50.65:34687)
0927 03:54:15.956966 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:15.956970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:15.956978 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:15.957008 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:15.957009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:15.957022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:15.957081 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:15.957387 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.958400 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.959842 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:15.960609 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:15.960667 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:15.960743 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:15.960994 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 03:54:15.961159 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":90,"threads_started":1}
I20260927 03:54:15.961916 9006 tablet_service.cc:1467] Processing CreateTablet for tablet 1895d9e952d9489b9785d9835c7ef18d (DEFAULT_TABLE table=test-table [id=a540e3c70d01471aa66b97022decf0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:15.961917 9146 tablet_service.cc:1467] Processing CreateTablet for tablet 1895d9e952d9489b9785d9835c7ef18d (DEFAULT_TABLE table=test-table [id=a540e3c70d01471aa66b97022decf0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:15.962324 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895d9e952d9489b9785d9835c7ef18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:15.962359 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895d9e952d9489b9785d9835c7ef18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:15.965265 9240 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Bootstrap starting.
I20260927 03:54:15.965857 9241 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Bootstrap starting.
I20260927 03:54:15.965914 9240 tablet_bootstrap.cc:654] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:15.966179 8866 tablet_service.cc:1467] Processing CreateTablet for tablet 1895d9e952d9489b9785d9835c7ef18d (DEFAULT_TABLE table=test-table [id=a540e3c70d01471aa66b97022decf0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:15.966315 9240 log.cc:824] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:15.966471 9241 tablet_bootstrap.cc:654] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:15.966492 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895d9e952d9489b9785d9835c7ef18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:15.966738 9241 log.cc:824] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:15.967113 9240 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: No bootstrap required, opened a new log
I20260927 03:54:15.967222 9240 ts_tablet_manager.cc:1397] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:54:15.967415 9241 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: No bootstrap required, opened a new log
I20260927 03:54:15.967478 9241 ts_tablet_manager.cc:1397] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:15.969530 9244 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Bootstrap starting.
I20260927 03:54:15.969507 9240 raft_consensus.cc:348] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.969393 9241 raft_consensus.cc:348] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.969658 9240 raft_consensus.cc:374] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:15.969707 9240 raft_consensus.cc:729] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Initialized, Role: FOLLOWER
I20260927 03:54:15.969710 9241 raft_consensus.cc:374] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:15.969748 9241 raft_consensus.cc:729] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Initialized, Role: FOLLOWER
I20260927 03:54:15.969831 9240 consensus_queue.cc:260] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.969857 9241 consensus_queue.cc:260] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.970142 9241 ts_tablet_manager.cc:1428] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:15.970144 9240 ts_tablet_manager.cc:1428] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:54:15.970261 9212 heartbeater.cc:503] Master 127.1.50.126:38053 was elected leader, sending a full tablet report...
I20260927 03:54:15.970315 9244 tablet_bootstrap.cc:654] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:15.970532 9072 heartbeater.cc:503] Master 127.1.50.126:38053 was elected leader, sending a full tablet report...
I20260927 03:54:15.970631 9244 log.cc:824] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:15.971487 9244 tablet_bootstrap.cc:492] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: No bootstrap required, opened a new log
I20260927 03:54:15.971560 9244 ts_tablet_manager.cc:1397] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:15.973057 9244 raft_consensus.cc:348] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.973193 9244 raft_consensus.cc:374] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:15.973214 9244 raft_consensus.cc:729] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Initialized, Role: FOLLOWER
I20260927 03:54:15.973333 9244 consensus_queue.cc:260] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:15.973568 9244 ts_tablet_manager.cc:1428] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:54:15.973601 8932 heartbeater.cc:503] Master 127.1.50.126:38053 was elected leader, sending a full tablet report...
W20260927 03:54:16.010143 9073 tablet.cc:2369] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:54:16.114065 8933 tablet.cc:2369] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:54:16.163056 9213 tablet.cc:2369] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:16.179646 9246 raft_consensus.cc:482] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:16.179792 9246 raft_consensus.cc:504] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.180176 9246 leader_election.cc:288] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687)
I20260927 03:54:16.183421 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.180376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:32805 (local address 127.1.50.67:44853)
0927 03:54:16.180429 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.180441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.180485 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.180662 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.180663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.180665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.180729 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.180731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.181470 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.181471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.182559 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.182561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.183183 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.183184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.183260 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.183264 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.183312 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.183347 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:54:16.183451 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.180467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:44853 (local address 127.1.50.66:32805)
0927 03:54:16.180536 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.180538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.180546 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:16.180573 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.180574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.180586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.180638 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.180816 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.181402 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.182639 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.183088 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.183152 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.183163 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.183332 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.183378 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:54:16.183485 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.180474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:55771 (local address 127.1.50.65:34687)
0927 03:54:16.180527 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.180529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.180824 (+ 295us) server_negotiation.cc:407] Connection header received
0927 03:54:16.180848 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.180849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.180860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.180905 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.181065 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.181652 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.182805 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.183165 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.183232 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.183241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.183387 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.183426 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:54:16.183525 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.180382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.1.50.67:55771)
0927 03:54:16.180736 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.180746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.180793 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.180917 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.180918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.180920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.180978 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.180979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.181745 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.181746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.182725 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.182727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.183246 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.183247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.183307 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.183312 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.183365 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.183401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":152,"thread_start_us":156,"threads_started":1}
I20260927 03:54:16.183892 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895d9e952d9489b9785d9835c7ef18d" candidate_uuid: "3477eae3be45442bb2361fab45fd2418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e15455fdf79488ead3de5b4424777ae" is_pre_election: true
I20260927 03:54:16.183954 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895d9e952d9489b9785d9835c7ef18d" candidate_uuid: "3477eae3be45442bb2361fab45fd2418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f" is_pre_election: true
I20260927 03:54:16.184046 8886 raft_consensus.cc:2454] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3477eae3be45442bb2361fab45fd2418 in term 0.
I20260927 03:54:16.184099 9026 raft_consensus.cc:2454] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3477eae3be45442bb2361fab45fd2418 in term 0.
I20260927 03:54:16.184305 9100 leader_election.cc:302] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3477eae3be45442bb2361fab45fd2418, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.184441 9246 raft_consensus.cc:2781] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:16.184507 9246 raft_consensus.cc:482] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:16.184523 9246 raft_consensus.cc:3037] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.185436 9246 raft_consensus.cc:504] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.185559 9246 leader_election.cc:288] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [CANDIDATE]: Term 1 election: Requested vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687)
I20260927 03:54:16.185714 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895d9e952d9489b9785d9835c7ef18d" candidate_uuid: "3477eae3be45442bb2361fab45fd2418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f"
I20260927 03:54:16.185720 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895d9e952d9489b9785d9835c7ef18d" candidate_uuid: "3477eae3be45442bb2361fab45fd2418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e15455fdf79488ead3de5b4424777ae"
I20260927 03:54:16.185789 9026 raft_consensus.cc:3037] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.185797 8886 raft_consensus.cc:3037] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.186506 9026 raft_consensus.cc:2454] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3477eae3be45442bb2361fab45fd2418 in term 1.
I20260927 03:54:16.186522 8886 raft_consensus.cc:2454] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3477eae3be45442bb2361fab45fd2418 in term 1.
I20260927 03:54:16.186705 9097 leader_election.cc:302] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3477eae3be45442bb2361fab45fd2418, 4514a7b408bb4389bf680047b5fd739f; no voters:
I20260927 03:54:16.186796 9246 raft_consensus.cc:2781] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:16.186957 9246 raft_consensus.cc:686] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 1 LEADER]: Becoming Leader. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Running, Role: LEADER
I20260927 03:54:16.187040 9246 consensus_queue.cc:237] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.187783 8729 catalog_manager.cc:5495] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3477eae3be45442bb2361fab45fd2418 (127.1.50.67). New cstate: current_term: 1 leader_uuid: "3477eae3be45442bb2361fab45fd2418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } health_report { overall_health: HEALTHY } } }
I20260927 03:54:16.242708 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.240129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.1.50.67:41021)
0927 03:54:16.240215 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.240217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.240227 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:16.240262 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.240263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.240282 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.240369 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:16.240558 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.241039 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.241948 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.242195 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.242262 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.242434 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.242539 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.242599 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.242612 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.242632 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 03:54:16.244529 9026 raft_consensus.cc:1261] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Refusing update from remote peer 3477eae3be45442bb2361fab45fd2418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.244527 8886 raft_consensus.cc:1261] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Refusing update from remote peer 3477eae3be45442bb2361fab45fd2418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.244850 9250 consensus_queue.cc:1035] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.244980 9246 consensus_queue.cc:1035] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.247256 9254 mvcc.cc:205] Tried to move back new op lower bound from 7333811225574977536 to 7333811225342619648. Current Snapshot: MvccSnapshot[applied={T|T < 7333811225574977536}]
I20260927 03:54:16.247543 9253 mvcc.cc:205] Tried to move back new op lower bound from 7333811225574977536 to 7333811225342619648. Current Snapshot: MvccSnapshot[applied={T|T < 7333811225574977536 or (T in {7333811225574977536})}]
I20260927 03:54:16.249406 8729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.249498 8729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.250195 8729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Sending DeleteTablet for 3 replicas of tablet 1895d9e952d9489b9785d9835c7ef18d
I20260927 03:54:16.250486 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet 1895d9e952d9489b9785d9835c7ef18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 03:54:16.250543 8866 tablet_service.cc:1514] Processing DeleteTablet for tablet 1895d9e952d9489b9785d9835c7ef18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:55974
I20260927 03:54:16.250703 9146 tablet_service.cc:1514] Processing DeleteTablet for tablet 1895d9e952d9489b9785d9835c7ef18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:35904
I20260927 03:54:16.250895 9264 tablet_replica.cc:330] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: stopping tablet replica
I20260927 03:54:16.250895 9263 tablet_replica.cc:330] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: stopping tablet replica
I20260927 03:54:16.250993 9263 raft_consensus.cc:2229] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.250993 9264 raft_consensus.cc:2229] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.250941 8729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:16.251070 8729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:16.251087 9264 raft_consensus.cc:2258] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.251087 9263 raft_consensus.cc:2258] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.251710 9264 ts_tablet_manager.cc:1905] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.252131 9263 ts_tablet_manager.cc:1905] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.252233 8866 tablet_service.cc:1467] Processing CreateTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 (DEFAULT_TABLE table=test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.252344 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dac97c2cb1411abcf7da1cd54e1460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.253347 9264 ts_tablet_manager.cc:1918] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.253423 9264 log.cc:1196] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/1895d9e952d9489b9785d9835c7ef18d
I20260927 03:54:16.253598 9244 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Bootstrap starting.
I20260927 03:54:16.253648 9264 ts_tablet_manager.cc:1939] T 1895d9e952d9489b9785d9835c7ef18d P 6e15455fdf79488ead3de5b4424777ae: Deleting consensus metadata
I20260927 03:54:16.253811 9265 tablet_replica.cc:330] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: stopping tablet replica
I20260927 03:54:16.253907 9265 raft_consensus.cc:2229] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:16.254005 9244 tablet_bootstrap.cc:654] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.254051 9265 raft_consensus.cc:2258] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.254105 9263 ts_tablet_manager.cc:1918] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.254158 9263 log.cc:1196] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/1895d9e952d9489b9785d9835c7ef18d
I20260927 03:54:16.254379 9263 ts_tablet_manager.cc:1939] T 1895d9e952d9489b9785d9835c7ef18d P 4514a7b408bb4389bf680047b5fd739f: Deleting consensus metadata
I20260927 03:54:16.254527 9265 ts_tablet_manager.cc:1905] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.254640 9006 tablet_service.cc:1467] Processing CreateTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 (DEFAULT_TABLE table=test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.254751 9244 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: No bootstrap required, opened a new log
I20260927 03:54:16.254756 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dac97c2cb1411abcf7da1cd54e1460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.254791 9244 ts_tablet_manager.cc:1397] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:16.254941 9244 raft_consensus.cc:348] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.255003 9244 raft_consensus.cc:374] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.255020 9244 raft_consensus.cc:729] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.255309 9244 consensus_queue.cc:260] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.255440 9244 ts_tablet_manager.cc:1428] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:16.255878 9241 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Bootstrap starting.
I20260927 03:54:16.255972 9265 ts_tablet_manager.cc:1918] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.256019 9265 log.cc:1196] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/1895d9e952d9489b9785d9835c7ef18d
I20260927 03:54:16.256165 9265 ts_tablet_manager.cc:1939] T 1895d9e952d9489b9785d9835c7ef18d P 3477eae3be45442bb2361fab45fd2418: Deleting consensus metadata
I20260927 03:54:16.256251 9241 tablet_bootstrap.cc:654] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.256814 9241 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: No bootstrap required, opened a new log
I20260927 03:54:16.256851 9241 ts_tablet_manager.cc:1397] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.256989 9241 raft_consensus.cc:348] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.257033 9241 raft_consensus.cc:374] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.257045 9241 raft_consensus.cc:729] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.257081 9241 consensus_queue.cc:260] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.257162 9241 ts_tablet_manager.cc:1428] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.257233 8716 catalog_manager.cc:4841] TS 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021): tablet 1895d9e952d9489b9785d9835c7ef18d (table test-table [id=a540e3c70d01471aa66b97022decf0f3]) successfully deleted
I20260927 03:54:16.257272 9146 tablet_service.cc:1467] Processing CreateTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 (DEFAULT_TABLE table=test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.257426 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dac97c2cb1411abcf7da1cd54e1460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.258169 8713 catalog_manager.cc:4841] TS 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805): tablet 1895d9e952d9489b9785d9835c7ef18d (table test-table [id=a540e3c70d01471aa66b97022decf0f3]) successfully deleted
I20260927 03:54:16.258381 8716 catalog_manager.cc:4841] TS 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687): tablet 1895d9e952d9489b9785d9835c7ef18d (table test-table [id=a540e3c70d01471aa66b97022decf0f3]) successfully deleted
I20260927 03:54:16.258642 9240 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Bootstrap starting.
I20260927 03:54:16.259047 9240 tablet_bootstrap.cc:654] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.259685 9240 tablet_bootstrap.cc:492] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: No bootstrap required, opened a new log
I20260927 03:54:16.259738 9240 ts_tablet_manager.cc:1397] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.259886 9240 raft_consensus.cc:348] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.259953 9240 raft_consensus.cc:374] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.259972 9240 raft_consensus.cc:729] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.260017 9240 consensus_queue.cc:260] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.260099 9240 ts_tablet_manager.cc:1428] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:16.353037 9248 raft_consensus.cc:482] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:16.353137 9248 raft_consensus.cc:504] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.353446 9248 leader_election.cc:288] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.357168 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.353803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:52403 (local address 127.1.50.67:41021)
0927 03:54:16.353845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.353847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.353856 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:16.353941 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.353942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.353951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.353977 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.354212 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.354712 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356136 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.356692 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356780 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.356792 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.357051 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.357098 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:54:16.357172 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.353685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:48041 (local address 127.1.50.66:32805)
0927 03:54:16.353731 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.353733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.353887 (+ 154us) server_negotiation.cc:407] Connection header received
0927 03:54:16.353975 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.353977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.353986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.354065 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.354239 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.354951 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356167 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.356698 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356878 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.356891 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.357047 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.357096 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:54:16.357380 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.353616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:32805 (local address 127.1.50.65:48041)
0927 03:54:16.353781 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.353792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.353913 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.354073 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.354074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.354076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.354147 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.354148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.355037 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.355038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.356092 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.356094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.356880 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.356963 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.356967 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.357254 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.357301 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":18,"thread_start_us":55,"threads_started":1}
I20260927 03:54:16.357750 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.353646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:41021 (local address 127.1.50.65:52403)
0927 03:54:16.353803 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.353813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.353862 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.354056 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.354057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.354059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.354128 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.354129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.354794 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.354795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.356039 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.356042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.356870 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.356871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.356952 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.356956 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.357614 (+ 658us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.357663 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":27,"thread_start_us":87,"threads_started":1}
I20260927 03:54:16.358081 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dac97c2cb1411abcf7da1cd54e1460" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f" is_pre_election: true
I20260927 03:54:16.358191 9026 raft_consensus.cc:2454] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:16.358392 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dac97c2cb1411abcf7da1cd54e1460" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418" is_pre_election: true
I20260927 03:54:16.358430 8817 leader_election.cc:302] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4514a7b408bb4389bf680047b5fd739f, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.358505 9166 raft_consensus.cc:2454] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:16.358542 9248 raft_consensus.cc:2781] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:16.358585 9248 raft_consensus.cc:482] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:16.358611 9248 raft_consensus.cc:3037] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.359241 9248 raft_consensus.cc:504] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.359373 9248 leader_election.cc:288] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Requested vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.359489 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dac97c2cb1411abcf7da1cd54e1460" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f"
I20260927 03:54:16.359553 9026 raft_consensus.cc:3037] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.359540 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dac97c2cb1411abcf7da1cd54e1460" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418"
I20260927 03:54:16.359606 9166 raft_consensus.cc:3037] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.360044 9026 raft_consensus.cc:2454] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:16.360204 8817 leader_election.cc:302] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4514a7b408bb4389bf680047b5fd739f, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.360280 9248 raft_consensus.cc:2781] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:16.360409 9166 raft_consensus.cc:2454] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:16.360416 9248 raft_consensus.cc:686] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 1 LEADER]: Becoming Leader. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Running, Role: LEADER
I20260927 03:54:16.360491 9248 consensus_queue.cc:237] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.361167 8729 catalog_manager.cc:5495] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e15455fdf79488ead3de5b4424777ae (127.1.50.65). New cstate: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:54:16 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481255, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.126@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:16.401286 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.394794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.1.50.126:38053)
0927 03:54:16.394863 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.394865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.394937 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:54:16.394966 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.394967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.394977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.395040 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:16.395509 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.396005 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.396721 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.396908 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.396961 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.398952 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:16.398954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:16.398954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:16.398960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:16.400334 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:16.400678 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:16.400679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:16.400679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:16.400721 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:16.400864 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:16.400865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:16.400865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:16.400962 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:16.401009 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.401113 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.401222 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":48}
I20260927 03:54:16.401273 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.394707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:38053 (local address 127.0.0.1:50512)
0927 03:54:16.394848 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.394858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.394917 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.395050 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.395051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.395334 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:16.395394 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.395397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.396092 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.396093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.396642 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.396644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.396968 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.396969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.397009 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.397516 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:16.397521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:16.398885 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:16.400407 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:16.400414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:16.400416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:16.400623 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:16.400764 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:16.400764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:16.400765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:16.400817 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:16.401005 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:16.401009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:16.401046 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.401093 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.401181 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260927 03:54:16.403100 8729 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:50512:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:54:16.403352 8729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:16.404758 9006 tablet_service.cc:1467] Processing CreateTablet for tablet 138d9034c80745b28db91bed17174c9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=174c593ce22a43e9b3f588f1c7c5de6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:54:16.404757 8866 tablet_service.cc:1467] Processing CreateTablet for tablet 138d9034c80745b28db91bed17174c9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=174c593ce22a43e9b3f588f1c7c5de6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:54:16.404776 9146 tablet_service.cc:1467] Processing CreateTablet for tablet 138d9034c80745b28db91bed17174c9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=174c593ce22a43e9b3f588f1c7c5de6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:54:16.404903 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138d9034c80745b28db91bed17174c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.404903 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138d9034c80745b28db91bed17174c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.404904 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138d9034c80745b28db91bed17174c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.405993 9241 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: Bootstrap starting.
I20260927 03:54:16.406370 9241 tablet_bootstrap.cc:654] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.406378 9240 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: Bootstrap starting.
I20260927 03:54:16.406679 9240 tablet_bootstrap.cc:654] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.406914 9244 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: Bootstrap starting.
I20260927 03:54:16.406985 9241 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: No bootstrap required, opened a new log
I20260927 03:54:16.407020 9241 ts_tablet_manager.cc:1397] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.407159 9241 raft_consensus.cc:348] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.407220 9241 raft_consensus.cc:374] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.407218 9240 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: No bootstrap required, opened a new log
I20260927 03:54:16.407241 9241 raft_consensus.cc:729] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.407253 9240 ts_tablet_manager.cc:1397] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.407300 9244 tablet_bootstrap.cc:654] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.407292 9241 consensus_queue.cc:260] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.407347 9240 raft_consensus.cc:348] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.407387 9241 ts_tablet_manager.cc:1428] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.407405 9240 raft_consensus.cc:374] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.407423 9240 raft_consensus.cc:729] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.407460 9240 consensus_queue.cc:260] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.407449 9245 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.407546 9240 ts_tablet_manager.cc:1428] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.407539 9245 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.407661 9246 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.407748 9246 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.407892 9244 tablet_bootstrap.cc:492] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: No bootstrap required, opened a new log
I20260927 03:54:16.407940 9244 ts_tablet_manager.cc:1397] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:16.408071 9244 raft_consensus.cc:348] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.408138 9244 raft_consensus.cc:374] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.408170 9244 raft_consensus.cc:729] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.408211 9244 consensus_queue.cc:260] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.408290 9244 ts_tablet_manager.cc:1428] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:16.408397 9248 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.408466 9248 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.472244 9248 raft_consensus.cc:482] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:16.472342 9248 raft_consensus.cc:504] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.472509 9248 leader_election.cc:288] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.472779 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138d9034c80745b28db91bed17174c9e" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f" is_pre_election: true
I20260927 03:54:16.472868 9026 raft_consensus.cc:2454] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:16.472851 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138d9034c80745b28db91bed17174c9e" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418" is_pre_election: true
I20260927 03:54:16.472962 9166 raft_consensus.cc:2454] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:16.473039 8817 leader_election.cc:302] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4514a7b408bb4389bf680047b5fd739f, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.473145 9248 raft_consensus.cc:2781] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:16.473193 9248 raft_consensus.cc:482] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:16.473212 9248 raft_consensus.cc:3037] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.473942 9248 raft_consensus.cc:504] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.474102 9248 leader_election.cc:288] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Requested vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.474272 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138d9034c80745b28db91bed17174c9e" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418"
I20260927 03:54:16.474291 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138d9034c80745b28db91bed17174c9e" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f"
I20260927 03:54:16.474352 9026 raft_consensus.cc:3037] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.474350 9166 raft_consensus.cc:3037] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.474874 9026 raft_consensus.cc:2454] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:16.475028 9166 raft_consensus.cc:2454] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:16.475034 8817 leader_election.cc:302] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4514a7b408bb4389bf680047b5fd739f, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.475116 9248 raft_consensus.cc:2781] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:16.475178 9248 raft_consensus.cc:686] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [term 1 LEADER]: Becoming Leader. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Running, Role: LEADER
I20260927 03:54:16.475229 9248 consensus_queue.cc:237] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.475493 9248 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: TxnStatusTablet state changed. Reason: New leader 6e15455fdf79488ead3de5b4424777ae. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.475562 9248 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:54:16.475782 9283 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:54:16.475899 9283 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:54:16.475865 8728 catalog_manager.cc:5495] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e15455fdf79488ead3de5b4424777ae (127.1.50.65). New cstate: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:16.485951 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.483050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.1.50.65:34687)
0927 03:54:16.483151 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.483154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.483165 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:16.483222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.483223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.483244 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.483323 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:16.483429 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.484182 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.484981 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.485284 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.485357 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.485547 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.485684 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.485757 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.485840 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.485865 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 03:54:16.485965 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.482962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.0.0.1:55982)
0927 03:54:16.483056 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.483069 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.483130 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.483325 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.483326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.483327 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:54:16.483381 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.483383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.484281 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.484282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.484898 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.484901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.485375 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.485376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.485443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.485458 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.485773 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.485784 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.485835 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.485885 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 03:54:16.487299 9026 raft_consensus.cc:1261] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.487327 9166 raft_consensus.cc:1261] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.487591 9271 consensus_queue.cc:1035] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.487746 9248 consensus_queue.cc:1035] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.488245 9245 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: TxnStatusTablet state changed. Reason: New leader 6e15455fdf79488ead3de5b4424777ae. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.488336 9245 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.488478 9246 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: TxnStatusTablet state changed. Reason: New leader 6e15455fdf79488ead3de5b4424777ae. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.488562 9246 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.489212 9271 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.489295 9271 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:54:16.489379 9271 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.489427 9271 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 6e15455fdf79488ead3de5b4424777ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:54:16.489380 9255 mvcc.cc:205] Tried to move back new op lower bound from 7333811226569678848 to 7333811226522791936. Current Snapshot: MvccSnapshot[applied={T|T < 7333811226569678848 or (T in {7333811226569678848})}]
I20260927 03:54:16.489629 9245 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.489701 9245 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 4514a7b408bb4389bf680047b5fd739f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.490589 9246 tablet_replica.cc:439] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } } }
I20260927 03:54:16.490689 9246 tablet_replica.cc:442] T 138d9034c80745b28db91bed17174c9e P 3477eae3be45442bb2361fab45fd2418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:54:16.493510 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.491159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.1.50.65:34687)
0927 03:54:16.491193 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.491195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.491201 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:54:16.491220 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.491221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.491231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.491280 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:16.491382 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.491912 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.492955 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.493152 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.493205 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.493347 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.493372 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.493401 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.493443 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.493459 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:54:16.498793 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.496040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:47131 (local address 127.1.50.65:34687)
0927 03:54:16.496095 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.496097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.496261 (+ 164us) server_negotiation.cc:407] Connection header received
0927 03:54:16.496311 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.496312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.496320 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.496369 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:16.496486 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.496976 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.498143 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.498323 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.498383 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.498548 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.498572 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.498603 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.498696 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.498710 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:54:16.498888 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.495980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.1.50.65:47131)
0927 03:54:16.496194 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.496205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.496264 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.496372 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.496372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.496374 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:54:16.496431 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.496433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.497050 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.497051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.498074 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.498077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.498394 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.498394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.498457 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.498474 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.498627 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.498628 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.498672 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.498713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":137,"threads_started":1}
I20260927 03:54:16.500917 9026 raft_consensus.cc:1261] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.500932 9166 raft_consensus.cc:1261] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.501113 9248 consensus_queue.cc:1035] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.501240 9291 consensus_queue.cc:1035] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.511668 8728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.511767 8728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.512319 8728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Sending DeleteTablet for 3 replicas of tablet 51dac97c2cb1411abcf7da1cd54e1460
I20260927 03:54:16.512537 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 03:54:16.512668 9263 tablet_replica.cc:330] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: stopping tablet replica
I20260927 03:54:16.512709 9146 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:35904
I20260927 03:54:16.512730 9263 raft_consensus.cc:2229] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.512773 9263 raft_consensus.cc:2258] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.512794 9265 tablet_replica.cc:330] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: stopping tablet replica
I20260927 03:54:16.512842 9265 raft_consensus.cc:2229] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.512871 9265 raft_consensus.cc:2258] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.513095 9263 ts_tablet_manager.cc:1905] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.513221 9265 ts_tablet_manager.cc:1905] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.514451 9263 ts_tablet_manager.cc:1918] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:54:16.514499 9263 log.cc:1196] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/51dac97c2cb1411abcf7da1cd54e1460
I20260927 03:54:16.514747 9263 ts_tablet_manager.cc:1939] T 51dac97c2cb1411abcf7da1cd54e1460 P 4514a7b408bb4389bf680047b5fd739f: Deleting consensus metadata
I20260927 03:54:16.514811 9265 ts_tablet_manager.cc:1918] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:54:16.514876 9265 log.cc:1196] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/51dac97c2cb1411abcf7da1cd54e1460
I20260927 03:54:16.515070 9265 ts_tablet_manager.cc:1939] T 51dac97c2cb1411abcf7da1cd54e1460 P 3477eae3be45442bb2361fab45fd2418: Deleting consensus metadata
I20260927 03:54:16.515249 8713 catalog_manager.cc:4841] TS 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805): tablet 51dac97c2cb1411abcf7da1cd54e1460 (table test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]) successfully deleted
I20260927 03:54:16.515324 8865 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dac97c2cb1411abcf7da1cd54e1460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:55974
I20260927 03:54:16.515427 9264 tablet_replica.cc:330] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: stopping tablet replica
I20260927 03:54:16.515486 9264 raft_consensus.cc:2229] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:16.515489 8716 catalog_manager.cc:4841] TS 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021): tablet 51dac97c2cb1411abcf7da1cd54e1460 (table test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]) successfully deleted
I20260927 03:54:16.515586 9264 raft_consensus.cc:2258] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.515954 9264 ts_tablet_manager.cc:1905] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.517423 9264 ts_tablet_manager.cc:1918] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:54:16.517468 9264 log.cc:1196] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/51dac97c2cb1411abcf7da1cd54e1460
I20260927 03:54:16.517664 9264 ts_tablet_manager.cc:1939] T 51dac97c2cb1411abcf7da1cd54e1460 P 6e15455fdf79488ead3de5b4424777ae: Deleting consensus metadata
I20260927 03:54:16.517980 8716 catalog_manager.cc:4841] TS 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687): tablet 51dac97c2cb1411abcf7da1cd54e1460 (table test-table [id=3f43c52e05da4bf494ed3a3a4e7c9f8c]) successfully deleted
Sep 27 03:54:16 dist-test-slave-96fg krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:54:16 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:16 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:54:16.541328 8778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:16.536127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.1.50.126:38053)
0927 03:54:16.536180 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.536182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.536312 (+ 130us) server_negotiation.cc:407] Connection header received
0927 03:54:16.536409 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.536411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.536423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.536470 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:16.537020 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.537529 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.538351 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.538537 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.538591 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.541246 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50524: BlockingRecv error: recv got EOF from 127.0.0.1:50524 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 03:54:16 dist-test-slave-96fg krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM
I20260927 03:54:16.549064 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.542089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.1.50.126:38053)
0927 03:54:16.542131 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.542133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.542163 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:54:16.542217 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.542218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.542228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.542290 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:16.542682 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.543399 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.544119 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.544444 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.544505 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.546724 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:16.546725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:16.546726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:16.546733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:16.548053 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:16.548357 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:16.548358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:16.548359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:16.548423 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:16.548586 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:16.548586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:16.548587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:16.548724 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:16.548821 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.548899 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.548973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:54:16.550660 8728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:16.550881 8728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:16.552201 8865 tablet_service.cc:1467] Processing CreateTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 (DEFAULT_TABLE table=test-table [id=582a8bd4a6da48899798e0d35bac586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.552362 8865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bd554106ab43d2b3b20ae4c43b2992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.553810 9244 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Bootstrap starting.
I20260927 03:54:16.553845 9006 tablet_service.cc:1467] Processing CreateTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 (DEFAULT_TABLE table=test-table [id=582a8bd4a6da48899798e0d35bac586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.553960 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bd554106ab43d2b3b20ae4c43b2992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.554164 9244 tablet_bootstrap.cc:654] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.554175 9146 tablet_service.cc:1467] Processing CreateTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 (DEFAULT_TABLE table=test-table [id=582a8bd4a6da48899798e0d35bac586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.554286 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bd554106ab43d2b3b20ae4c43b2992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.555480 9240 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Bootstrap starting.
I20260927 03:54:16.555779 9240 tablet_bootstrap.cc:654] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.556246 9240 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: No bootstrap required, opened a new log
I20260927 03:54:16.556289 9240 ts_tablet_manager.cc:1397] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.556394 9240 raft_consensus.cc:348] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.556454 9240 raft_consensus.cc:374] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.556473 9240 raft_consensus.cc:729] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.556515 9240 consensus_queue.cc:260] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.556593 9240 ts_tablet_manager.cc:1428] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.556653 9241 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Bootstrap starting.
I20260927 03:54:16.557061 9241 tablet_bootstrap.cc:654] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.557730 9241 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: No bootstrap required, opened a new log
I20260927 03:54:16.557782 9241 ts_tablet_manager.cc:1397] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.557946 9241 raft_consensus.cc:348] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.558074 9244 tablet_bootstrap.cc:492] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: No bootstrap required, opened a new log
I20260927 03:54:16.558123 9244 ts_tablet_manager.cc:1397] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:54:16.558213 9241 raft_consensus.cc:374] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.558266 9241 raft_consensus.cc:729] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.558295 9244 raft_consensus.cc:348] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.558311 9241 consensus_queue.cc:260] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.558365 9244 raft_consensus.cc:374] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.558384 9244 raft_consensus.cc:729] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.558409 9241 ts_tablet_manager.cc:1428] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:16.558424 9244 consensus_queue.cc:260] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.558502 9244 ts_tablet_manager.cc:1428] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.892455 9245 raft_consensus.cc:482] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:16.892562 9245 raft_consensus.cc:504] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.892864 9245 leader_election.cc:288] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.896219 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.893107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.1.50.66:54681)
0927 03:54:16.893237 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.893250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.893308 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.893514 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.893514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.893517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.893573 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.893576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.894439 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.894440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.895329 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.895332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.895912 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.895913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.895991 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.895995 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.896065 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.896109 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":29,"thread_start_us":62,"threads_started":1}
I20260927 03:54:16.896210 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.893237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54681 (local address 127.1.50.65:34687)
0927 03:54:16.893291 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.893293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.893305 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:54:16.893423 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.893424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.893435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.893493 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.893670 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.894351 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.895424 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.895821 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.895874 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.895883 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.896098 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.896139 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:54:16.896426 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.893281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:41021 (local address 127.1.50.66:52895)
0927 03:54:16.893457 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:16.893466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:16.893530 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:16.893742 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:16.893743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:16.893746 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:16.893849 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.893851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.894609 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.894610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.895449 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:16.895451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.896126 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.896127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:16.896218 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.896222 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:54:16.896288 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:54:16.896336 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":19,"thread_start_us":44,"threads_started":1}
I20260927 03:54:16.896453 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.893367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:52895 (local address 127.1.50.67:41021)
0927 03:54:16.893574 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.893576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.893583 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:16.893632 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.893633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.893644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.893704 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:16.893941 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.894511 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.895550 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.896037 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.896099 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.896110 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.896304 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.896386 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":52,"threads_started":1}
I20260927 03:54:16.896692 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bd554106ab43d2b3b20ae4c43b2992" candidate_uuid: "4514a7b408bb4389bf680047b5fd739f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418" is_pre_election: true
I20260927 03:54:16.896780 9166 raft_consensus.cc:2454] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4514a7b408bb4389bf680047b5fd739f in term 0.
I20260927 03:54:16.896788 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bd554106ab43d2b3b20ae4c43b2992" candidate_uuid: "4514a7b408bb4389bf680047b5fd739f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e15455fdf79488ead3de5b4424777ae" is_pre_election: true
I20260927 03:54:16.896872 8886 raft_consensus.cc:2454] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4514a7b408bb4389bf680047b5fd739f in term 0.
I20260927 03:54:16.896961 8960 leader_election.cc:302] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3477eae3be45442bb2361fab45fd2418, 4514a7b408bb4389bf680047b5fd739f; no voters:
I20260927 03:54:16.897145 9245 raft_consensus.cc:2781] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:16.897197 9245 raft_consensus.cc:482] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:16.897241 9245 raft_consensus.cc:3037] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.897935 9245 raft_consensus.cc:504] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.898085 9245 leader_election.cc:288] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [CANDIDATE]: Term 1 election: Requested vote from peers 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:16.898272 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bd554106ab43d2b3b20ae4c43b2992" candidate_uuid: "4514a7b408bb4389bf680047b5fd739f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e15455fdf79488ead3de5b4424777ae"
I20260927 03:54:16.898344 8886 raft_consensus.cc:3037] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.898304 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bd554106ab43d2b3b20ae4c43b2992" candidate_uuid: "4514a7b408bb4389bf680047b5fd739f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418"
I20260927 03:54:16.898377 9166 raft_consensus.cc:3037] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:16.898967 8886 raft_consensus.cc:2454] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4514a7b408bb4389bf680047b5fd739f in term 1.
I20260927 03:54:16.899048 9166 raft_consensus.cc:2454] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4514a7b408bb4389bf680047b5fd739f in term 1.
I20260927 03:54:16.899153 8960 leader_election.cc:302] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4514a7b408bb4389bf680047b5fd739f, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:16.899250 9245 raft_consensus.cc:2781] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:16.899353 9245 raft_consensus.cc:686] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 1 LEADER]: Becoming Leader. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Running, Role: LEADER
I20260927 03:54:16.899415 9245 consensus_queue.cc:237] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.900115 8728 catalog_manager.cc:5495] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4514a7b408bb4389bf680047b5fd739f (127.1.50.66). New cstate: current_term: 1 leader_uuid: "4514a7b408bb4389bf680047b5fd739f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:16.915850 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:16.913252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.1.50.66:32805)
0927 03:54:16.913352 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:16.913353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:16.913359 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:54:16.913393 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:16.913393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:16.913415 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:16.913492 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:16.913632 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.914249 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.914981 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:16.915195 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:16.915268 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:16.915433 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:16.915565 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:16.915621 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:16.915738 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:54:16.915759 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260927 03:54:16.916921 9166 raft_consensus.cc:1261] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Refusing update from remote peer 4514a7b408bb4389bf680047b5fd739f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.916921 8886 raft_consensus.cc:1261] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Refusing update from remote peer 4514a7b408bb4389bf680047b5fd739f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:16.917227 9245 consensus_queue.cc:1035] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.917387 9313 consensus_queue.cc:1035] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:16.920501 8728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.920603 8728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:16.921192 8728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Sending DeleteTablet for 3 replicas of tablet 49bd554106ab43d2b3b20ae4c43b2992
I20260927 03:54:16.921458 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 03:54:16.921556 8865 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:55974
I20260927 03:54:16.921656 9263 tablet_replica.cc:330] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: stopping tablet replica
I20260927 03:54:16.921708 9263 raft_consensus.cc:2229] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:16.921702 8728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 03:54:16.921797 9263 raft_consensus.cc:2258] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:54:16.921857 8728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:16.921875 9146 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bd554106ab43d2b3b20ae4c43b2992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:16 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:35904
I20260927 03:54:16.921962 9264 tablet_replica.cc:330] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: stopping tablet replica
I20260927 03:54:16.922053 9264 raft_consensus.cc:2229] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.922104 9264 raft_consensus.cc:2258] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.922174 9265 tablet_replica.cc:330] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: stopping tablet replica
I20260927 03:54:16.922227 9265 raft_consensus.cc:2229] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:16.922279 9265 raft_consensus.cc:2258] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:16.922550 9264 ts_tablet_manager.cc:1905] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.922976 9265 ts_tablet_manager.cc:1905] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.924185 9264 ts_tablet_manager.cc:1918] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.924245 9264 log.cc:1196] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/49bd554106ab43d2b3b20ae4c43b2992
I20260927 03:54:16.924427 9264 ts_tablet_manager.cc:1939] T 49bd554106ab43d2b3b20ae4c43b2992 P 6e15455fdf79488ead3de5b4424777ae: Deleting consensus metadata
I20260927 03:54:16.924887 9263 ts_tablet_manager.cc:1905] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:16.924947 9006 tablet_service.cc:1467] Processing CreateTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc (DEFAULT_TABLE table=test-table [id=59947c2a48c04a89854018894bf8a4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.924934 8865 tablet_service.cc:1467] Processing CreateTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc (DEFAULT_TABLE table=test-table [id=59947c2a48c04a89854018894bf8a4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.925028 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bff6978efd40148ec3238f7a8cfcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.925046 8865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bff6978efd40148ec3238f7a8cfcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.926354 9241 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Bootstrap starting.
I20260927 03:54:16.926390 9263 ts_tablet_manager.cc:1918] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.926425 9263 log.cc:1196] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/49bd554106ab43d2b3b20ae4c43b2992
I20260927 03:54:16.926543 9244 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: Bootstrap starting.
I20260927 03:54:16.926615 9263 ts_tablet_manager.cc:1939] T 49bd554106ab43d2b3b20ae4c43b2992 P 4514a7b408bb4389bf680047b5fd739f: Deleting consensus metadata
I20260927 03:54:16.926739 9241 tablet_bootstrap.cc:654] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.926839 9244 tablet_bootstrap.cc:654] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.926894 9265 ts_tablet_manager.cc:1918] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:16.926949 9265 log.cc:1196] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/49bd554106ab43d2b3b20ae4c43b2992
I20260927 03:54:16.927168 9265 ts_tablet_manager.cc:1939] T 49bd554106ab43d2b3b20ae4c43b2992 P 3477eae3be45442bb2361fab45fd2418: Deleting consensus metadata
I20260927 03:54:16.927381 9146 tablet_service.cc:1467] Processing CreateTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc (DEFAULT_TABLE table=test-table [id=59947c2a48c04a89854018894bf8a4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:16.927465 8716 catalog_manager.cc:4841] TS 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021): tablet 49bd554106ab43d2b3b20ae4c43b2992 (table test-table [id=582a8bd4a6da48899798e0d35bac586f]) successfully deleted
I20260927 03:54:16.927493 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bff6978efd40148ec3238f7a8cfcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:16.927745 8713 catalog_manager.cc:4841] TS 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805): tablet 49bd554106ab43d2b3b20ae4c43b2992 (table test-table [id=582a8bd4a6da48899798e0d35bac586f]) successfully deleted
I20260927 03:54:16.927956 9241 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: No bootstrap required, opened a new log
I20260927 03:54:16.928000 9241 ts_tablet_manager.cc:1397] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:54:16.928114 9241 raft_consensus.cc:348] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.928165 9241 raft_consensus.cc:374] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.928177 9241 raft_consensus.cc:729] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4514a7b408bb4389bf680047b5fd739f, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.928216 9241 consensus_queue.cc:260] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.928295 9241 ts_tablet_manager.cc:1428] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:16.928903 9240 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Bootstrap starting.
I20260927 03:54:16.929199 9240 tablet_bootstrap.cc:654] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:16.929574 8716 catalog_manager.cc:4841] TS 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687): tablet 49bd554106ab43d2b3b20ae4c43b2992 (table test-table [id=582a8bd4a6da48899798e0d35bac586f]) successfully deleted
I20260927 03:54:16.929901 9240 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: No bootstrap required, opened a new log
I20260927 03:54:16.929952 9240 ts_tablet_manager.cc:1397] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:16.930096 9240 raft_consensus.cc:348] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.930233 9240 raft_consensus.cc:374] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.930254 9240 raft_consensus.cc:729] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3477eae3be45442bb2361fab45fd2418, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.930447 9240 consensus_queue.cc:260] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.930554 9240 ts_tablet_manager.cc:1428] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:16.930678 9244 tablet_bootstrap.cc:492] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: No bootstrap required, opened a new log
I20260927 03:54:16.930727 9244 ts_tablet_manager.cc:1397] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:54:16.930845 9244 raft_consensus.cc:348] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.930909 9244 raft_consensus.cc:374] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:16.930928 9244 raft_consensus.cc:729] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Initialized, Role: FOLLOWER
I20260927 03:54:16.930964 9244 consensus_queue.cc:260] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:16.931048 9244 ts_tablet_manager.cc:1428] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:17.034988 9271 raft_consensus.cc:482] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:17.035121 9271 raft_consensus.cc:504] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:17.035308 9271 leader_election.cc:288] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:17.035558 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bff6978efd40148ec3238f7a8cfcbc" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418" is_pre_election: true
I20260927 03:54:17.035554 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bff6978efd40148ec3238f7a8cfcbc" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f" is_pre_election: true
I20260927 03:54:17.035661 9026 raft_consensus.cc:2454] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:17.035661 9166 raft_consensus.cc:2454] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 0.
I20260927 03:54:17.035807 8820 leader_election.cc:302] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3477eae3be45442bb2361fab45fd2418, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:17.035917 9271 raft_consensus.cc:2781] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:17.035960 9271 raft_consensus.cc:482] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:17.035976 9271 raft_consensus.cc:3037] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:17.036707 9271 raft_consensus.cc:504] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:17.036829 9271 leader_election.cc:288] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Requested vote from peers 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805), 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021)
I20260927 03:54:17.036991 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bff6978efd40148ec3238f7a8cfcbc" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4514a7b408bb4389bf680047b5fd739f"
I20260927 03:54:17.037001 9166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bff6978efd40148ec3238f7a8cfcbc" candidate_uuid: "6e15455fdf79488ead3de5b4424777ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3477eae3be45442bb2361fab45fd2418"
I20260927 03:54:17.037076 9166 raft_consensus.cc:3037] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:17.037076 9026 raft_consensus.cc:3037] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:17.037782 9166 raft_consensus.cc:2454] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:17.037782 9026 raft_consensus.cc:2454] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e15455fdf79488ead3de5b4424777ae in term 1.
I20260927 03:54:17.037971 8820 leader_election.cc:302] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3477eae3be45442bb2361fab45fd2418, 6e15455fdf79488ead3de5b4424777ae; no voters:
I20260927 03:54:17.038151 9271 raft_consensus.cc:2781] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:17.038264 9271 raft_consensus.cc:686] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 1 LEADER]: Becoming Leader. State: Replica: 6e15455fdf79488ead3de5b4424777ae, State: Running, Role: LEADER
I20260927 03:54:17.038321 9271 consensus_queue.cc:237] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } }
I20260927 03:54:17.038903 8727 catalog_manager.cc:5495] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e15455fdf79488ead3de5b4424777ae (127.1.50.65). New cstate: current_term: 1 leader_uuid: "6e15455fdf79488ead3de5b4424777ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e15455fdf79488ead3de5b4424777ae" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:17.050889 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:17.048884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.1.50.65:34687)
0927 03:54:17.048923 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:17.048925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:17.048936 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:17.048993 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:17.048994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:17.049003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:17.049044 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:17.049121 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:17.049587 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:17.050386 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:17.050566 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:17.050617 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:17.050708 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:17.050733 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:17.050764 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:17.050800 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:54:17.050817 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:54:17.052870 9026 raft_consensus.cc:1261] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:17.052870 9166 raft_consensus.cc:1261] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Refusing update from remote peer 6e15455fdf79488ead3de5b4424777ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:17.053107 9271 consensus_queue.cc:1035] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3477eae3be45442bb2361fab45fd2418" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:17.053247 9326 consensus_queue.cc:1035] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4514a7b408bb4389bf680047b5fd739f" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:17.062683 8728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:17.062793 8728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:17.063361 8728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb02728e397478293b7faf3292787b2: Sending DeleteTablet for 3 replicas of tablet b1bff6978efd40148ec3238f7a8cfcbc
I20260927 03:54:17.063561 8865 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:17 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:55974
I20260927 03:54:17.063602 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:17 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 03:54:17.063707 9146 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bff6978efd40148ec3238f7a8cfcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:17 UTC) from {username='oryx', principal='oryx/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:35904
I20260927 03:54:17.063798 9264 tablet_replica.cc:330] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: stopping tablet replica
I20260927 03:54:17.063799 9265 tablet_replica.cc:330] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: stopping tablet replica
I20260927 03:54:17.063853 9264 raft_consensus.cc:2229] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:17.063858 9265 raft_consensus.cc:2229] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:17.063892 9265 raft_consensus.cc:2258] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:17.063915 9264 raft_consensus.cc:2258] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:17.063985 9263 tablet_replica.cc:330] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: stopping tablet replica
I20260927 03:54:17.064050 9263 raft_consensus.cc:2229] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:17.064090 9263 raft_consensus.cc:2258] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:17.064461 9263 ts_tablet_manager.cc:1905] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:17.065595 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8801
I20260927 03:54:17.065734 9264 ts_tablet_manager.cc:1905] T b1bff6978efd40148ec3238f7a8cfcbc P 6e15455fdf79488ead3de5b4424777ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:17.066672 9263 ts_tablet_manager.cc:1918] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:54:17.066731 9263 log.cc:1196] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/b1bff6978efd40148ec3238f7a8cfcbc
I20260927 03:54:17.066912 9263 ts_tablet_manager.cc:1939] T b1bff6978efd40148ec3238f7a8cfcbc P 4514a7b408bb4389bf680047b5fd739f: Deleting consensus metadata
I20260927 03:54:17.067257 8713 catalog_manager.cc:4841] TS 4514a7b408bb4389bf680047b5fd739f (127.1.50.66:32805): tablet b1bff6978efd40148ec3238f7a8cfcbc (table test-table [id=59947c2a48c04a89854018894bf8a4ae]) successfully deleted
I20260927 03:54:17.071866 9265 ts_tablet_manager.cc:1905] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:54:17.071925 8716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:34687 (error 108)
I20260927 03:54:17.072428 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:17.072265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34687 (local address 127.0.0.1:56010)
0927 03:54:17.072345 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:17.072387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":36,"threads_started":1}
I20260927 03:54:17.072527 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8941
I20260927 03:54:17.073417 9265 ts_tablet_manager.cc:1918] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:54:17.073473 9265 log.cc:1196] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/b1bff6978efd40148ec3238f7a8cfcbc
I20260927 03:54:17.073721 9265 ts_tablet_manager.cc:1939] T b1bff6978efd40148ec3238f7a8cfcbc P 3477eae3be45442bb2361fab45fd2418: Deleting consensus metadata
W20260927 03:54:17.074004 8716 catalog_manager.cc:4584] TS 6e15455fdf79488ead3de5b4424777ae (127.1.50.65:34687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1bff6978efd40148ec3238f7a8cfcbc: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34687: connect: Connection refused (error 111)
I20260927 03:54:17.074153 8716 catalog_manager.cc:4841] TS 3477eae3be45442bb2361fab45fd2418 (127.1.50.67:41021): tablet b1bff6978efd40148ec3238f7a8cfcbc (table test-table [id=59947c2a48c04a89854018894bf8a4ae]) successfully deleted
I20260927 03:54:17.077569 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9081
I20260927 03:54:17.082119 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 8697
2026-09-27T03:54:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2870 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:17 dist-test-slave-96fg krb5kdc[9336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:17 dist-test-slave-96fg krb5kdc[9336](info): set up 2 sockets
Sep 27 03:54:17 dist-test-slave-96fg krb5kdc[9336](info): commencing operation
krb5kdc: starting...
W20260927 03:54:18.042524 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:18 dist-test-slave-96fg krb5kdc[9336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:18.157466 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:42967
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:18.239840 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:18.240067 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:18.240100 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:18.241452 9349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:18.241503 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:18.241523 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:18.241539 9349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:18.241555 9349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:18.243337 9349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:42967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:18.243613 9349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:18.243834 9349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:18.248564 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:18.248562 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:18.248600 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:18.248795 9349 server_base.cc:1034] running on GCE node
I20260927 03:54:18.249035 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:18.249323 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:18.250461 9349 hybrid_clock.cc:648] HybridClock initialized: now 1790481258250447 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T03:54:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1178 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:18 dist-test-slave-96fg krb5kdc[9362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:18 dist-test-slave-96fg krb5kdc[9362](info): set up 2 sockets
Sep 27 03:54:18 dist-test-slave-96fg krb5kdc[9362](info): commencing operation
krb5kdc: starting...
W20260927 03:54:19.218706 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:19.350060 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35029
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:19.438472 9375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:19.438730 9375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:19.438762 9375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:19.440027 9375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:19.440075 9375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:19.440094 9375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:19.440110 9375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:19.440125 9375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:19.441771 9375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:19.442063 9375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:19.442310 9375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:19.447691 9383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.447700 9380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.447659 9381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:19.447919 9375 server_base.cc:1034] running on GCE node
I20260927 03:54:19.448259 9375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:19.448583 9375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:19.449738 9375 hybrid_clock.cc:648] HybridClock initialized: now 1790481259449716 us; error 47 us; skew 500 ppm
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:19.457644 9375 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:19.458261 9375 webserver.cc:460] Webserver started at http://127.1.50.126:39893/ using document root <none> and password file <none>
I20260927 03:54:19.458537 9375 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:19.458590 9375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:19.458732 9375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:19.459960 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "9b6bda682fa8433ba22d527f17f70a9f"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "726261f2da4f3ff29869d8477c627f4c"
server_key_iv: "4daa4dd053e2cf293958274650fa97ff"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.460327 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "9b6bda682fa8433ba22d527f17f70a9f"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "726261f2da4f3ff29869d8477c627f4c"
server_key_iv: "4daa4dd053e2cf293958274650fa97ff"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.461884 9375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:19.462872 9390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.463115 9375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:19.463217 9375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9b6bda682fa8433ba22d527f17f70a9f"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "726261f2da4f3ff29869d8477c627f4c"
server_key_iv: "4daa4dd053e2cf293958274650fa97ff"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.463280 9375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:19.472748 9375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:19.475046 9375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:19.475193 9375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:19.480250 9375 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35029
I20260927 03:54:19.480351 9442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35029 every 8 connection(s)
I20260927 03:54:19.480741 9375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:19.481742 9443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:19.484531 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Bootstrap starting.
I20260927 03:54:19.485240 9443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:19.485561 9443 log.cc:824] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:19.485831 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9375
I20260927 03:54:19.485942 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:19.486167 1225 external_mini_cluster.cc:1428] Setting key 58484bd8f06515d8b243f26d56485566
I20260927 03:54:19.486407 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: No bootstrap required, opened a new log
I20260927 03:54:19.488742 9443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:19.488936 9443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:19.488976 9443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6bda682fa8433ba22d527f17f70a9f, State: Initialized, Role: FOLLOWER
I20260927 03:54:19.489117 9443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:19.489207 9443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:19.489240 9443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:19.489292 9443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:19.490141 9443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:19.490258 9443 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6bda682fa8433ba22d527f17f70a9f; no voters:
I20260927 03:54:19.490437 9443 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:19.490525 9448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:19.490725 9443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:19.490828 9448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 LEADER]: Becoming Leader. State: Replica: 9b6bda682fa8433ba22d527f17f70a9f, State: Running, Role: LEADER
I20260927 03:54:19.491010 9448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:19.491547 9448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6bda682fa8433ba22d527f17f70a9f. Latest consensus state: current_term: 1 leader_uuid: "9b6bda682fa8433ba22d527f17f70a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } } }
I20260927 03:54:19.491740 9448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:19.492214 9458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:19.492929 9458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:19.493541 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b6bda682fa8433ba22d527f17f70a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } } }
I20260927 03:54:19.494683 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:19.495299 9458 catalog_manager.cc:1347] Generated new cluster ID: d5074012f0ad4ec3aab1c1663e064825
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:19.495345 9458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:19.499975 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:19.486845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.1.50.126:35029)
0927 03:54:19.487086 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:19.487100 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:19.487145 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:54:19.487918 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:19.487930 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:19.488003 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:19.488189 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:19.488863 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.490710 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.491907 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.492616 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.492734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:19.495881 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:19.495887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:19.495887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:19.495903 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:19.498154 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.498572 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.498576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.498576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.498694 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.498889 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.498890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.498890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.499261 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:19.499455 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:19.499553 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:54:19.499722 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260927 03:54:19.518651 9458 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:19.519563 9458 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:19.531545 9458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Generated new TSK 0
I20260927 03:54:19.531811 9458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:54:19.562626 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:19.648175 9469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:19.648401 9469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:19.648434 9469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:19.650135 9469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:19.650182 9469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:19.650223 9469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:19.651732 9469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:19.651988 9469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:19.652204 9469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:19.657552 9477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.657631 9475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.657554 9474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:19.657635 9469 server_base.cc:1034] running on GCE node
I20260927 03:54:19.657992 9469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:19.658305 9469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:19.659487 9469 hybrid_clock.cc:648] HybridClock initialized: now 1790481259659464 us; error 48 us; skew 500 ppm
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:19.662290 9469 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:19.662776 9469 webserver.cc:460] Webserver started at http://127.1.50.65:33995/ using document root <none> and password file <none>
I20260927 03:54:19.662995 9469 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:19.663045 9469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:19.663172 9469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:19.664268 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "6eb5d4b1b768439e912b48b62eed7c7a"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "9c639a0f1a447a5094fc26ecea418730"
server_key_iv: "0d7600f9b53048191ad2ba6785ad11dd"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.664600 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "6eb5d4b1b768439e912b48b62eed7c7a"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "9c639a0f1a447a5094fc26ecea418730"
server_key_iv: "0d7600f9b53048191ad2ba6785ad11dd"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.665994 9469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:19.667013 9484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.667351 9469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:19.667464 9469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "6eb5d4b1b768439e912b48b62eed7c7a"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "9c639a0f1a447a5094fc26ecea418730"
server_key_iv: "0d7600f9b53048191ad2ba6785ad11dd"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.667546 9469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:19.684353 9469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:19.686506 9469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:19.686626 9469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:19.686884 9469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:19.687175 9469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:19.687212 9469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.687242 9469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:19.687258 9469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.693907 9469 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:37425
I20260927 03:54:19.694010 9597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:37425 every 8 connection(s)
I20260927 03:54:19.694319 9469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:19.697898 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9469
I20260927 03:54:19.698055 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:19.698256 1225 external_mini_cluster.cc:1428] Setting key b649b025306e507abed60cc6c06bad1a
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:19.704835 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:19.695110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37075 (local address 127.1.50.126:35029)
0927 03:54:19.695179 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:19.695181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:19.695436 (+ 255us) server_negotiation.cc:407] Connection header received
0927 03:54:19.695958 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:19.695960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:19.695972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:19.696023 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:19.696953 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.697473 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.698792 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.699090 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.699156 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:19.701855 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:19.701857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:19.701858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:19.701866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:19.703276 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.703768 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.703769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.703769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.703820 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.704094 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.704095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.704095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.704217 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:19.704268 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:19.704652 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 03:54:19.704755 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:54:19.704988 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:19.695086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:37075)
0927 03:54:19.695304 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:19.695345 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:19.695862 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:19.696095 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:19.696100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:19.696654 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:19.696864 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:19.696868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.697557 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.697558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:19.698656 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:19.698658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.699178 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.699179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:19.699335 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:19.699918 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:19.699927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:19.701711 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:19.703365 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:19.703371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:19.703373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:19.703663 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:19.703885 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:19.703886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:19.703887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:19.704022 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:19.704280 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:19.704287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:19.704458 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:54:19.704587 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 03:54:19.704754 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260927 03:54:19.705749 9598 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35029
I20260927 03:54:19.705857 9598 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:19.706100 9598 heartbeater.cc:511] Master 127.1.50.126:35029 requested a full tablet report, sending...
I20260927 03:54:19.706779 9407 ts_manager.cc:194] Registered new tserver with Master: 6eb5d4b1b768439e912b48b62eed7c7a (127.1.50.65:37425)
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:54:19.707549 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:37075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:19.726053 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:19.813787 9603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:19.814069 9603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:19.814107 9603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:19.815369 9603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:19.815407 9603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:19.815441 9603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:19.816938 9603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:19.817190 9603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:19.817418 9603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:19.822990 9609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.822969 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:19.823077 9603 server_base.cc:1034] running on GCE node
W20260927 03:54:19.822969 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:19.823459 9603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:19.823720 9603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:19.824878 9603 hybrid_clock.cc:648] HybridClock initialized: now 1790481259824856 us; error 44 us; skew 500 ppm
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:19.827732 9603 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:19.828297 9603 webserver.cc:460] Webserver started at http://127.1.50.66:44831/ using document root <none> and password file <none>
I20260927 03:54:19.828541 9603 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:19.828594 9603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:19.828733 9603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:19.829937 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "522851018c1c483aa42239af0748be28"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "85b2cbfb8e5291b9f7efbcba227257c4"
server_key_iv: "e22339facafedbfd89d9f54e3ac682c3"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.830379 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "522851018c1c483aa42239af0748be28"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "85b2cbfb8e5291b9f7efbcba227257c4"
server_key_iv: "e22339facafedbfd89d9f54e3ac682c3"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.831872 9603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:19.832757 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.833055 9603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:19.833161 9603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "522851018c1c483aa42239af0748be28"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "85b2cbfb8e5291b9f7efbcba227257c4"
server_key_iv: "e22339facafedbfd89d9f54e3ac682c3"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.833236 9603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:19.849428 9603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:19.851817 9603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:19.851963 9603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:19.852216 9603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:19.852559 9603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:19.852599 9603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.852631 9603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:19.852655 9603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.859900 9603 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:40551
I20260927 03:54:19.859956 9731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:40551 every 8 connection(s)
I20260927 03:54:19.860314 9603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:19.861891 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9603
I20260927 03:54:19.861984 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:19.862219 1225 external_mini_cluster.cc:1428] Setting key af98e1d1a478bb93ddc5969008587dee
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:19.870746 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:19.861176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:33767 (local address 127.1.50.126:35029)
0927 03:54:19.861286 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:19.861289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:19.861531 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:54:19.862282 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:19.862284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:19.862296 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:19.862346 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:19.863513 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.864005 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.865005 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.865206 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.865260 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:19.867963 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:19.867965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:19.867966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:19.867975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:19.869295 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.869705 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.869707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.869707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.869760 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:19.869981 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:19.869982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:19.869982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:19.870145 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:19.870208 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:19.870555 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:54:19.870645 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:54:19.870960 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:19.861094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:33767)
0927 03:54:19.861352 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:19.861421 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:19.862171 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:19.862438 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:19.862443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:19.863191 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:19.863420 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:19.863424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.864085 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.864086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:19.864927 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:19.864929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:19.865277 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:19.865278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:19.865419 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:19.865985 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:19.865996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:19.867850 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:19.869381 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:19.869389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:19.869392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:19.869639 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:19.869815 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:19.869815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:19.869816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:19.869926 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:19.870204 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:19.870211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:19.870394 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 03:54:19.870522 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:54:19.870712 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260927 03:54:19.871817 9732 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35029
I20260927 03:54:19.871923 9732 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:19.872130 9732 heartbeater.cc:511] Master 127.1.50.126:35029 requested a full tablet report, sending...
I20260927 03:54:19.872565 9407 ts_manager.cc:194] Registered new tserver with Master: 522851018c1c483aa42239af0748be28 (127.1.50.66:40551)
I20260927 03:54:19.873133 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:33767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:19.889657 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:19.974974 9737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:19.975191 9737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:19.975221 9737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:19.976409 9737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:19.976449 9737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:19.976482 9737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:19.977883 9737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:19.978139 9737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:19.978384 9737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:19.983860 9743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.983877 9745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:19.983877 9742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:19.984380 9737 server_base.cc:1034] running on GCE node
I20260927 03:54:19.984745 9737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:19.985116 9737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:19.986315 9737 hybrid_clock.cc:648] HybridClock initialized: now 1790481259986292 us; error 37 us; skew 500 ppm
Sep 27 03:54:19 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:19.989369 9737 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:19.989899 9737 webserver.cc:460] Webserver started at http://127.1.50.67:39677/ using document root <none> and password file <none>
I20260927 03:54:19.990185 9737 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:19.990242 9737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:19.990496 9737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:19.992060 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "3a8f28921de3418ea54996bd8475e542"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "dd5c9f66a6019748adf4d981f457317b"
server_key_iv: "c3240e9b14321906a281a0b02e6b247e"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.992489 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "3a8f28921de3418ea54996bd8475e542"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "dd5c9f66a6019748adf4d981f457317b"
server_key_iv: "c3240e9b14321906a281a0b02e6b247e"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.994204 9737 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 03:54:19.995316 9752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:19.995633 9737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:19.995743 9737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "3a8f28921de3418ea54996bd8475e542"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "dd5c9f66a6019748adf4d981f457317b"
server_key_iv: "c3240e9b14321906a281a0b02e6b247e"
server_key_version: "encryptionkey@0"
I20260927 03:54:19.995817 9737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:20.012943 9737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:20.015394 9737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:20.015551 9737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:20.015873 9737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:20.016214 9737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:20.016248 9737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.016283 9737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:20.016306 9737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.023821 9737 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:33901
I20260927 03:54:20.023902 9865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:33901 every 8 connection(s)
I20260927 03:54:20.024269 9737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:20.025883 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9737
I20260927 03:54:20.025985 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:20.026213 1225 external_mini_cluster.cc:1428] Setting key f776b54c8c2bbd6287def3abde7d1b51
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:20.035189 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.025101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:37791 (local address 127.1.50.126:35029)
0927 03:54:20.025194 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.025196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.025435 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:54:20.025994 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.025996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.026011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.026135 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.027410 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.027956 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.029247 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.029459 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.029603 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.032134 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:20.032137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:20.032138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:20.032147 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:20.033542 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:20.034076 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:20.034078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:20.034078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:20.034146 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:20.034449 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:20.034450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:20.034450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:20.034599 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:20.034669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:20.035017 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:54:20.035105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 03:54:20.035332 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.025077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:37791)
0927 03:54:20.025260 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.025304 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.025854 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.026258 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.026262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.027046 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.027315 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.027322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.028075 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.028076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.029134 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.029137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.029626 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.029627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.029762 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.030354 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.030363 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.031996 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:20.033657 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:20.033663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:20.033665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:20.033928 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:20.034243 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:20.034244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:20.034244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:20.034356 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:20.034682 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:20.034687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:20.034849 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 03:54:20.034947 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:54:20.035090 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260927 03:54:20.035977 9866 heartbeater.cc:347] Connected to a master server at 127.1.50.126:35029
I20260927 03:54:20.036113 9866 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:20.036375 9866 heartbeater.cc:511] Master 127.1.50.126:35029 requested a full tablet report, sending...
I20260927 03:54:20.036888 9407 ts_manager.cc:194] Registered new tserver with Master: 3a8f28921de3418ea54996bd8475e542 (127.1.50.67:33901)
I20260927 03:54:20.037289 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:37791
I20260927 03:54:20.038318 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
Entry for principal oryx/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab.
WARNING: no policy specified for oryx/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
Entry for principal oryx/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab.
WARNING: no policy specified for oryx/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
Entry for principal oryx/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab.
I20260927 03:54:20.117069 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9469
I20260927 03:54:20.121302 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9603
I20260927 03:54:20.126204 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9737
I20260927 03:54:20.130995 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9375
I20260927 03:54:20.137305 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35029
--webserver_interface=127.1.50.126
--webserver_port=39893
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:20.220012 9875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:20.220242 9875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:20.220264 9875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:20.221439 9875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:20.221488 9875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:20.221503 9875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:20.221513 9875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:20.221524 9875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:20.223026 9875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:20.223232 9875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:20.223420 9875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:20.228379 9880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.228423 9883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.228667 9875 server_base.cc:1034] running on GCE node
W20260927 03:54:20.228392 9881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.229024 9875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:20.229331 9875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:20.230489 9875 hybrid_clock.cc:648] HybridClock initialized: now 1790481260230436 us; error 73 us; skew 500 ppm
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481260, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:20.233342 9875 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:20.234076 9875 webserver.cc:460] Webserver started at http://127.1.50.126:39893/ using document root <none> and password file <none>
I20260927 03:54:20.234368 9875 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:20.234440 9875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:20.236136 9875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:20.237023 9890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.237279 9875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:20.237385 9875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9b6bda682fa8433ba22d527f17f70a9f"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "726261f2da4f3ff29869d8477c627f4c"
server_key_iv: "4daa4dd053e2cf293958274650fa97ff"
server_key_version: "encryptionkey@0"
I20260927 03:54:20.237843 9875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:20.258131 9875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:20.260679 9875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:20.260803 9875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:20.265076 9875 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35029
I20260927 03:54:20.265141 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35029 every 8 connection(s)
I20260927 03:54:20.265506 9875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:20.267349 9943 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:54:20.268081 9943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Bootstrap starting.
I20260927 03:54:20.273416 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9875
I20260927 03:54:20.273828 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:37425
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:35029
--webserver_port=33995
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:54:20.279058 9943 log.cc:824] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:20.281273 9943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:54:20.281625 9943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Bootstrap complete.
I20260927 03:54:20.284073 9943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:20.284351 9943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6bda682fa8433ba22d527f17f70a9f, State: Initialized, Role: FOLLOWER
I20260927 03:54:20.284524 9943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:20.284597 9943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:20.284636 9943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:20.284690 9943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:54:20.285857 9943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:20.285959 9943 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6bda682fa8433ba22d527f17f70a9f; no voters:
I20260927 03:54:20.286202 9943 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:54:20.286258 9947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:54:20.286391 9947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [term 2 LEADER]: Becoming Leader. State: Replica: 9b6bda682fa8433ba22d527f17f70a9f, State: Running, Role: LEADER
I20260927 03:54:20.286489 9947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } }
I20260927 03:54:20.286594 9943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:20.286826 9947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6bda682fa8433ba22d527f17f70a9f. Latest consensus state: current_term: 2 leader_uuid: "9b6bda682fa8433ba22d527f17f70a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } } }
I20260927 03:54:20.286988 9947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:20.287309 9949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b6bda682fa8433ba22d527f17f70a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bda682fa8433ba22d527f17f70a9f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35029 } } }
I20260927 03:54:20.287432 9949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:20.287519 9953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:20.288187 9953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:20.288693 9953 catalog_manager.cc:1259] Loaded cluster ID: d5074012f0ad4ec3aab1c1663e064825
I20260927 03:54:20.288739 9953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:20.289635 9953 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:20.289824 9953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b6bda682fa8433ba22d527f17f70a9f: Loaded TSK: 0
I20260927 03:54:20.290079 9953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:54:20.373804 9945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:20.374011 9945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:20.374064 9945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:20.375247 9945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:20.375291 9945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:20.375326 9945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:20.376751 9945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.65.keytab
--principal=oryx/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=33995
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:20.376986 9945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:20.377192 9945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:20.382182 9970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.382179 9945 server_base.cc:1034] running on GCE node
W20260927 03:54:20.382141 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.382146 9972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.382619 9945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:20.382894 9945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:20.384048 9945 hybrid_clock.cc:648] HybridClock initialized: now 1790481260384034 us; error 31 us; skew 500 ppm
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481260, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:20.386587 9945 init.cc:377] Logged in from keytab as oryx/127.1.50.65@KRBTEST.COM (short username oryx)
I20260927 03:54:20.387090 9945 webserver.cc:460] Webserver started at http://127.1.50.65:33995/ using document root <none> and password file <none>
I20260927 03:54:20.387315 9945 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:20.387373 9945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:20.388883 9945 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:20.389575 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.389770 9945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:20.389865 9945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "6eb5d4b1b768439e912b48b62eed7c7a"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "9c639a0f1a447a5094fc26ecea418730"
server_key_iv: "0d7600f9b53048191ad2ba6785ad11dd"
server_key_version: "encryptionkey@0"
I20260927 03:54:20.390301 9945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:20.402433 9945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:20.404487 9945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:20.404601 9945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:20.404820 9945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:20.405151 9945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:20.405185 9945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.405213 9945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:20.405238 9945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.410724 9945 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:37425
I20260927 03:54:20.410763 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:37425 every 8 connection(s)
I20260927 03:54:20.411095 9945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.420086 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.411874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:37669)
0927 03:54:20.412032 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.412072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.412517 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.412984 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.412988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.413545 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.413728 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.413732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.415403 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.415404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.416207 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.416209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.417013 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.417014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.417120 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.417603 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.417611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.419935 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260927 03:54:20.420306 10093 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:35029 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.50.126:35029: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
W20260927 03:54:20.420552 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.411945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37669 (local address 127.1.50.126:35029)
0927 03:54:20.412132 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.412140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.412175 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:54:20.412709 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.412716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.412769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.412919 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.413854 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.415284 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.416307 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.416922 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.417034 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.420105 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:37669: BlockingRecv error: recv got EOF from 127.1.50.65:37669 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260927 03:54:20.420629 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 9945
I20260927 03:54:20.420912 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:40551
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:35029
--webserver_port=44831
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.425616 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.420570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:50573)
0927 03:54:20.421100 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.421111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.421164 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.421303 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.421304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.421452 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.421510 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.421511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.422412 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.422413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.423153 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.423155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.423479 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.423480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.423538 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.423938 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.423943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.425549 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511}
W20260927 03:54:20.425689 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.420670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:50573 (local address 127.1.50.126:35029)
0927 03:54:20.420736 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.420738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.421162 (+ 424us) server_negotiation.cc:407] Connection header received
0927 03:54:20.421213 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.421214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.421227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.421285 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.421571 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.422330 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.423230 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.423421 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.423476 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.425641 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:50573: BlockingRecv error: recv got EOF from 127.1.50.65:50573 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.429620 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.425913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:44279)
0927 03:54:20.425963 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.425974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.426009 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.426164 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.426165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.426283 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.426310 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.426311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.426886 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.426887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.427370 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.427372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.427676 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.427677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.427723 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.428099 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.428104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.429554 (+ 1450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:54:20.429792 10093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:54:20.429919 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.425960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44279 (local address 127.1.50.126:35029)
0927 03:54:20.425995 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.425996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.426004 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:20.426087 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.426088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.426098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.426149 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.426370 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.426824 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.427433 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.427625 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.427674 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.429636 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:44279: BlockingRecv error: recv got EOF from 127.1.50.65:44279 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:54:20.509384 10097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:20.509588 10097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:20.509617 10097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:20.510818 10097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:20.510859 10097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:20.510885 10097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:20.512341 10097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.66.keytab
--principal=oryx/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:40551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=44831
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:20.512562 10097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:20.512794 10097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:20.517555 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.517542 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.517544 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.517786 10097 server_base.cc:1034] running on GCE node
I20260927 03:54:20.517999 10097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:20.518262 10097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:20.519410 10097 hybrid_clock.cc:648] HybridClock initialized: now 1790481260519393 us; error 33 us; skew 500 ppm
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481260, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:20.521780 10097 init.cc:377] Logged in from keytab as oryx/127.1.50.66@KRBTEST.COM (short username oryx)
I20260927 03:54:20.522274 10097 webserver.cc:460] Webserver started at http://127.1.50.66:44831/ using document root <none> and password file <none>
I20260927 03:54:20.522485 10097 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:20.522547 10097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:20.523896 10097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:20.524556 10112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.524722 10097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:20.524827 10097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "522851018c1c483aa42239af0748be28"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "85b2cbfb8e5291b9f7efbcba227257c4"
server_key_iv: "e22339facafedbfd89d9f54e3ac682c3"
server_key_version: "encryptionkey@0"
I20260927 03:54:20.525164 10097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:20.534159 10097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:20.536181 10097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:20.536326 10097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:20.536526 10097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:20.536806 10097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:20.536836 10097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.536864 10097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:20.536885 10097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.542538 10097 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:40551
I20260927 03:54:20.542585 10225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:40551 every 8 connection(s)
I20260927 03:54:20.542897 10097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:20.546386 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10097
I20260927 03:54:20.546645 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:33901
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:35029
--webserver_port=39677
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:54:20.550062 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.543525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:47301 (local address 127.1.50.126:35029)
0927 03:54:20.543583 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.543585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.543794 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:54:20.544284 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.544285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.544299 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.544352 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.545235 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.545718 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.546714 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.546964 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.547020 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.549987 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:47301: BlockingRecv error: recv got EOF from 127.1.50.66:47301 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:54:20.550107 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.543525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:47301)
0927 03:54:20.543678 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.543719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.544181 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.544416 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.544420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.544957 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.545141 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.545146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.545800 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.545800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.546630 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.546633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.547036 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.547037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.547162 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.547696 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.547756 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.549891 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
W20260927 03:54:20.550508 10226 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:35029 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.50.126:35029: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.554735 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.550730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:38993)
0927 03:54:20.550780 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.550790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.550833 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.550951 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.550952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.551073 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.551114 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.551116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.551681 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.551681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.552168 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.552169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.552463 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.552464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.552521 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.552950 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.552953 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.554622 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:54:20.554755 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.550822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:38993 (local address 127.1.50.126:35029)
0927 03:54:20.550862 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.550863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.550870 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:20.550890 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.550891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.550900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.550944 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.551175 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.551617 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.552234 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.552411 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.552463 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.554707 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:38993: BlockingRecv error: recv got EOF from 127.1.50.66:38993 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.558883 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.555145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:56983)
0927 03:54:20.555189 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.555199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.555239 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.555340 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.555341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.555452 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.555478 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.555479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.556046 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.556047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.556522 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.556524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.556814 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.556815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.556860 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.557242 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.557246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.558814 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:54:20.558950 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:56983 (local address 127.1.50.126:35029)
0927 03:54:20.555252 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.555254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.555261 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:20.555282 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.555282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.555291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.555334 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.555536 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.555980 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.556587 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.556760 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.556810 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.558902 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:56983: BlockingRecv error: recv got EOF from 127.1.50.66:56983 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 03:54:20.559279 10226 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:54:20.627121 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:20.627348 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:20.627377 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:20.628626 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:20.628669 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:20.628705 10229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:20.630162 10229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/krb5kdc/oryx_127.1.50.67.keytab
--principal=oryx/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:33901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=39677
--tserver_master_addrs=127.1.50.126:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:20.630388 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:20.630581 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:20.635398 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.635411 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:20.635407 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:20.635550 10229 server_base.cc:1034] running on GCE node
I20260927 03:54:20.635860 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:20.636086 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:20.637241 10229 hybrid_clock.cc:648] HybridClock initialized: now 1790481260637225 us; error 29 us; skew 500 ppm
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481260, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:20.639761 10229 init.cc:377] Logged in from keytab as oryx/127.1.50.67@KRBTEST.COM (short username oryx)
I20260927 03:54:20.640223 10229 webserver.cc:460] Webserver started at http://127.1.50.67:39677/ using document root <none> and password file <none>
I20260927 03:54:20.640447 10229 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:20.640504 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:20.641855 10229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:20.642524 10244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.642756 10229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:54:20.642843 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "3a8f28921de3418ea54996bd8475e542"
format_stamp: "Formatted at 2026-09-27 03:54:19 on dist-test-slave-96fg"
server_key: "dd5c9f66a6019748adf4d981f457317b"
server_key_iv: "c3240e9b14321906a281a0b02e6b247e"
server_key_version: "encryptionkey@0"
I20260927 03:54:20.643177 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:20.658193 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:20.660353 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:20.660493 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:20.660692 10229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:20.660975 10229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:20.661008 10229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.661031 10229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:20.661052 10229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:20.666810 10229 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:33901
I20260927 03:54:20.666873 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:33901 every 8 connection(s)
I20260927 03:54:20.667176 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:20.671463 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10229
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:54:20.675398 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.667949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:53433 (local address 127.1.50.126:35029)
0927 03:54:20.668025 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.668027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.668189 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:54:20.668688 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.668690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.668702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.668773 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.669750 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.670303 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.671279 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.671547 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.671617 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.675334 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:53433: BlockingRecv error: recv got EOF from 127.1.50.67:53433 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:54:20.675731 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.667920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:53433)
0927 03:54:20.668067 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.668108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.668568 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.668835 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.668838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.669407 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.669595 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.669600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.670380 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.670381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.671177 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.671180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.671631 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.671632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.671771 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.672732 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.672742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.675509 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
W20260927 03:54:20.675962 10358 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:35029 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.50.126:35029: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.681932 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.672061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.1.50.126:35029)
0927 03:54:20.672170 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.672172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.672178 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:54:20.672196 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.672197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.672206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.672259 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.673014 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.673658 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.674940 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.675287 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.675348 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.677197 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:20.677202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:20.677203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:20.677243 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:20.680231 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:20.680612 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:20.680616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:20.680616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:20.680731 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:20.680966 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:20.680967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:20.680967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:20.681271 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:20.681430 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:20.681714 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:54:20.681837 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260927 03:54:20.682358 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.676320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:52199)
0927 03:54:20.676353 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.676363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.676426 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.676571 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.676572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.676714 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.676775 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.676777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.677747 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.677748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.678566 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.678569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.679190 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.679191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.679262 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.680055 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.680060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.682305 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 03:54:20.682423 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.676391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:52199 (local address 127.1.50.126:35029)
0927 03:54:20.676432 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.676433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.676468 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:54:20.676489 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.676490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.676500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.676547 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.676953 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.677663 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.678649 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.679076 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.679134 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.682378 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:52199: BlockingRecv error: recv got EOF from 127.1.50.67:52199 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:20 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:20.686854 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:20.682605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:47727)
0927 03:54:20.682653 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:20.682664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:20.682705 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:20.682909 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:20.682910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:20.683010 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:20.683039 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.683040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.683633 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.683634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.684114 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:20.684116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.684525 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.684526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:20.684578 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.684974 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:20.684979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:20.686788 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:54:20.686924 10096 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:20.682694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:47727 (local address 127.1.50.126:35029)
0927 03:54:20.682728 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:20.682729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:20.682736 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:20.682756 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:20.682757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:20.682766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:20.682868 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:20.683108 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.683566 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.684185 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:20.684448 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:20.684508 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:20.686872 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:47727: BlockingRecv error: recv got EOF from 127.1.50.67:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:54:20.687005 10358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:21.435459 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:21.430241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:36797)
0927 03:54:21.430430 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:21.430444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:21.430534 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:21.430742 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:21.430743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:21.430906 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:21.430972 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.430974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.431778 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.431780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.432382 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.432384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.432783 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.432784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.432832 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.433313 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:21.433317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:21.435376 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
W20260927 03:54:21.435606 10364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:21.430354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:36797 (local address 127.1.50.126:35029)
0927 03:54:21.430494 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:21.430496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:21.430506 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:21.430592 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:21.430592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:21.430603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:21.430667 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:21.431079 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.431664 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.432500 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.432715 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.432772 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.435515 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:36797: BlockingRecv error: recv got EOF from 127.1.50.65:36797 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:21.564991 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:21.559798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:54087)
0927 03:54:21.559976 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:21.559990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:21.560094 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:21.560249 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:21.560250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:21.560496 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:21.560564 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.560566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.561274 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.561275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.561919 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.561921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.562294 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.562295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.562352 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.562983 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:21.562988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:21.564906 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
W20260927 03:54:21.565056 10364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:21.559939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54087 (local address 127.1.50.126:35029)
0927 03:54:21.560014 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:21.560016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:21.560085 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:54:21.560155 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:21.560156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:21.560167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:21.560227 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:21.560652 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.561168 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.561998 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.562206 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.562290 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.565002 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:54087: BlockingRecv error: recv got EOF from 127.1.50.66:54087 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:21 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:21.692619 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:21.687461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:36981)
0927 03:54:21.687639 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:21.687653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:21.687740 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:21.688059 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:21.688060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:21.688252 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:21.688305 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.688307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.689076 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.689077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.689656 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:21.689659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.690067 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.690068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:21.690114 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.690585 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:21.690589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:21.692541 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
W20260927 03:54:21.692847 10364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:21.687529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:36981 (local address 127.1.50.126:35029)
0927 03:54:21.687611 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:21.687613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:21.687731 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:54:21.687882 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:21.687883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:21.687900 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:21.687990 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:21.688421 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.688982 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.689766 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:21.689964 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:21.690089 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:21.692713 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:36981: BlockingRecv error: recv got EOF from 127.1.50.67:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:22.440788 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:22.436006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:43237)
0927 03:54:22.436135 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:22.436145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:22.436192 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:22.436413 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:22.436414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:22.436565 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:22.436611 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.436613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.437331 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.437331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.437882 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.437884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.438345 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.438346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.438398 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.438899 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:22.438904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:22.440718 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
W20260927 03:54:22.440951 10368 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:22.436122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:43237 (local address 127.1.50.126:35029)
0927 03:54:22.436250 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:22.436252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:22.436263 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:22.436291 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:22.436291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:22.436302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:22.436377 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:22.436704 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.437242 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.437982 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.438242 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.438304 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.440886 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:43237: BlockingRecv error: recv got EOF from 127.1.50.65:43237 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:22.570312 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:22.565550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:45227)
0927 03:54:22.565674 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:22.565684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:22.565740 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:22.565905 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:22.565906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:22.566091 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:22.566148 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.566150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.566879 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.566880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.567424 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.567426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.567838 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.567839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.567889 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.568334 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:22.568339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:22.570243 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
W20260927 03:54:22.570416 10368 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:22.565671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:45227 (local address 127.1.50.126:35029)
0927 03:54:22.565732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:22.565786 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:22.565793 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:22.565816 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:22.565816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:22.565826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:22.565881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:22.566249 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.566797 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.567510 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.567744 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.567811 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.570348 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:45227: BlockingRecv error: recv got EOF from 127.1.50.66:45227 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:22 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:22.698204 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:22.693227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:54173)
0927 03:54:22.693410 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:22.693423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:22.693515 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:22.693671 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:22.693672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:22.693852 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:22.693899 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.693901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.694651 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.694652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.695213 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:22.695215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.695595 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.695596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:22.695648 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.696153 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:22.696157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:22.698126 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
W20260927 03:54:22.698309 10368 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:22.693412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:54173 (local address 127.1.50.126:35029)
0927 03:54:22.693524 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:22.693526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:22.693540 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:54:22.693570 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:22.693571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:22.693584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:22.693641 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:22.693982 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.694562 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.695310 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:22.695507 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:22.695563 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:22.698239 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:54173: BlockingRecv error: recv got EOF from 127.1.50.67:54173 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:23.446430 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:23.441356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:52717)
0927 03:54:23.441510 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:23.441524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:23.441617 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:23.441812 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:23.441813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:23.442060 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:23.442149 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.442151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.442877 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.442877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.443545 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.443547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.443909 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.443909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.443953 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.444440 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:23.444445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:23.446316 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
W20260927 03:54:23.446506 10372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:23.441439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:52717 (local address 127.1.50.126:35029)
0927 03:54:23.441556 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:23.441558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:23.441608 (+ 50us) server_negotiation.cc:407] Connection header received
0927 03:54:23.441673 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:23.441673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:23.441712 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:23.441791 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:23.442268 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.442781 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.443637 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.443844 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.443899 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.446437 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:52717: BlockingRecv error: recv got EOF from 127.1.50.65:52717 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:23.575687 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:23.570863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:35733)
0927 03:54:23.571035 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:23.571046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:23.571111 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:23.571234 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:23.571235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:23.571425 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:23.571471 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.571473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.572107 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.572107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.572643 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.572645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.572968 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.572969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.573021 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.573471 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:23.573475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:23.575596 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
W20260927 03:54:23.575780 10372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:23.570967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:35733 (local address 127.1.50.126:35029)
0927 03:54:23.571022 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:23.571024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:23.571096 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:54:23.571163 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:23.571164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:23.571175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:23.571224 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:23.571546 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.572034 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.572719 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.572907 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.572963 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.575703 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:35733: BlockingRecv error: recv got EOF from 127.1.50.66:35733 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:23 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:23.704389 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:23.698967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:50125)
0927 03:54:23.699136 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:23.699147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:23.699201 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:23.699391 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:23.699392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:23.699553 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:23.699604 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.699605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.700375 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.700376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.701029 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:23.701033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.701522 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.701523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:23.701580 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.702106 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:23.702112 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:23.704263 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20260927 03:54:23.704453 10372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:23.699113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:50125 (local address 127.1.50.126:35029)
0927 03:54:23.699180 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:23.699182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:23.699193 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:23.699269 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:23.699270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:23.699286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:23.699371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:23.699687 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.700264 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.701161 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:23.701380 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:23.701492 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:23.704389 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:50125: BlockingRecv error: recv got EOF from 127.1.50.67:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:24.452382 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:24.447094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:53471)
0927 03:54:24.447247 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:24.447259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:24.447322 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:24.447570 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:24.447572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:24.447769 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:24.447835 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.447837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.448628 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.448629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.449249 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.449252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.449632 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.449633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.449724 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.450303 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:24.450309 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:24.452321 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
W20260927 03:54:24.452517 10376 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:24.447173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:53471 (local address 127.1.50.126:35029)
0927 03:54:24.447330 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:24.447331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:24.447344 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:54:24.447377 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:24.447377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:24.447389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:24.447499 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:24.447957 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.448490 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.449341 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.449561 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.449632 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.452436 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:53471: BlockingRecv error: recv got EOF from 127.1.50.65:53471 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:24.584699 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:24.576283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:40715)
0927 03:54:24.576432 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:24.576445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:24.576523 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:24.576664 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:24.576665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:24.576848 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:24.576896 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.576897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.577543 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.577545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.578143 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.578146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.578492 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.578493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.578543 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.582657 (+ 4114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:24.582662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:24.584635 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260927 03:54:24.584810 10376 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:24.576391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:40715 (local address 127.1.50.126:35029)
0927 03:54:24.576448 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:24.576450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:24.576508 (+ 58us) server_negotiation.cc:407] Connection header received
0927 03:54:24.576585 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:24.576586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:24.576599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:24.576649 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:24.576963 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.577467 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.578236 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.578432 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.578486 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.584749 (+ 6263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:40715: BlockingRecv error: recv got EOF from 127.1.50.66:40715 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:24 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:24.709617 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:24.704962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:46043)
0927 03:54:24.705084 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:24.705094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:24.705149 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:24.705321 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:24.705322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:24.705482 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:24.705529 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.705531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.706254 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.706255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.706831 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:24.706834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.707163 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.707164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:24.707199 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.707638 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:24.707642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:24.709556 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
W20260927 03:54:24.709712 10376 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:24.705026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:46043 (local address 127.1.50.126:35029)
0927 03:54:24.705109 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:24.705110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:24.705137 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:54:24.705208 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:24.705209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:24.705220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:24.705276 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:24.705602 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.706139 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.706923 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:24.707106 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:24.707183 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:24.709655 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:46043: BlockingRecv error: recv got EOF from 127.1.50.67:46043 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:25.457252 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:25.452877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:58631)
0927 03:54:25.453022 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:25.453032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:25.453090 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:25.453214 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:25.453215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:25.453372 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:25.453420 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.453422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.454071 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.454072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.454602 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.454604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.454948 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.454949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.455002 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.455407 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:25.455411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:25.457180 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
W20260927 03:54:25.457401 10380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:25.452940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:58631 (local address 127.1.50.126:35029)
0927 03:54:25.453044 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:25.453046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:25.453073 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:54:25.453131 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:25.453132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:25.453140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:25.453200 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:25.453468 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.453991 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.454659 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.454867 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.454914 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.457331 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:58631: BlockingRecv error: recv got EOF from 127.1.50.65:58631 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:25.590637 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:25.585300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:54627)
0927 03:54:25.585488 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:25.585537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:25.585611 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:25.585892 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:25.585893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:25.586140 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:25.586207 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.586210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.586967 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.586969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.587587 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.587590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.588063 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.588064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.588120 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.588625 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:25.588630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:25.590560 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
W20260927 03:54:25.590791 10380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:25.585449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54627 (local address 127.1.50.126:35029)
0927 03:54:25.585534 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:25.585536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:25.585721 (+ 185us) server_negotiation.cc:407] Connection header received
0927 03:54:25.585759 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:25.585760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:25.585773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:25.585836 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:25.586285 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.586853 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.587726 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.587961 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.588030 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.590719 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:54627: BlockingRecv error: recv got EOF from 127.1.50.66:54627 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:25 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:25.715719 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:25.710362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:37323)
0927 03:54:25.710608 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:25.710620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:25.710711 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:25.710944 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:25.710945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:25.711155 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:25.711212 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.711214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.712055 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.712056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.712672 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:25.712674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.713078 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.713079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:25.713142 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.713687 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:25.713695 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:25.715653 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
W20260927 03:54:25.715888 10380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:25.710446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:37323 (local address 127.1.50.126:35029)
0927 03:54:25.710498 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:25.710500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:25.710691 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:54:25.710783 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:25.710784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:25.710804 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:25.710864 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:25.711373 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.711955 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.712775 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:25.712986 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:25.713058 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:25.715808 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:37323: BlockingRecv error: recv got EOF from 127.1.50.67:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:26.463070 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:26.457766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:36219)
0927 03:54:26.457957 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:26.457973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:26.458108 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:26.458268 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:26.458269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:26.458454 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:26.458505 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.458506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.459257 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.459258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.459812 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.459814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.460223 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.460224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.460267 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.460777 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:26.460780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:26.462994 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
W20260927 03:54:26.463251 10384 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:26.457897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:36219 (local address 127.1.50.126:35029)
0927 03:54:26.458106 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:26.458109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:26.458121 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:54:26.458152 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:26.458152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:26.458164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:26.458234 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:26.458594 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.459166 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.459926 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.460151 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.460212 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.463158 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:36219: BlockingRecv error: recv got EOF from 127.1.50.65:36219 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:26.595534 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:26.591084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:36521)
0927 03:54:26.591230 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:26.591242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:26.591317 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:26.591452 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:26.591453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:26.591621 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:26.591673 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.591675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.592306 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.592307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.592843 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.592845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.593187 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.593188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.593237 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.593714 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:26.593718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:26.595456 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260927 03:54:26.595597 10384 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:26.591158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:36521 (local address 127.1.50.126:35029)
0927 03:54:26.591240 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:26.591243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:26.591303 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:54:26.591377 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:26.591378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:26.591389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:26.591440 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:26.591739 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.592238 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.592914 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.593114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.593171 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.595544 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:36521: BlockingRecv error: recv got EOF from 127.1.50.66:36521 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:26 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:26.720789 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:26.716232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:45145)
0927 03:54:26.716376 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:26.716387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:26.716444 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:26.716556 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:26.716557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:26.716719 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:26.716762 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.716764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.717387 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.717388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.717925 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:26.717928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.718284 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.718285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:26.718335 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.718872 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:26.718876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:26.720725 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260927 03:54:26.720878 10384 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:26.716311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:45145 (local address 127.1.50.126:35029)
0927 03:54:26.716368 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:26.716370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:26.716439 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:54:26.716495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:26.716496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:26.716507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:26.716544 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:26.716817 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.717303 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.718009 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:26.718215 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:26.718266 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:26.720814 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:45145: BlockingRecv error: recv got EOF from 127.1.50.67:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:27.468175 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:27.463590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:39539)
0927 03:54:27.463765 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:27.463779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:27.463837 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:27.463966 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:27.463967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:27.464143 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:27.464191 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.464192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.464911 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.464911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.465462 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.465464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.465921 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.465922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.465974 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.466432 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:27.466436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:27.468103 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
W20260927 03:54:27.468286 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:27.463745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:39539 (local address 127.1.50.126:35029)
0927 03:54:27.463850 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:27.463852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:27.463861 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:27.463885 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:27.463886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:27.463896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:27.463951 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:27.464268 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.464830 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.465558 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.465829 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.465892 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.468208 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:39539: BlockingRecv error: recv got EOF from 127.1.50.65:39539 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":43,"threads_started":1}
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:27.600406 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:27.596043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:48377)
0927 03:54:27.596165 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:27.596176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:27.596227 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:27.596445 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:27.596447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:27.596601 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:27.596645 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.596646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.597348 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.597349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.597907 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.597908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.598288 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.598289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.598340 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.598732 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:27.598736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:27.600331 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260927 03:54:27.600485 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:27.596140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:48377 (local address 127.1.50.126:35029)
0927 03:54:27.596248 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:27.596250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:27.596262 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:54:27.596293 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:27.596294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:27.596312 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:27.596384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:27.596740 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.597273 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.597989 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.598231 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.598316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.600434 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:48377: BlockingRecv error: recv got EOF from 127.1.50.66:48377 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:27 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:27.725864 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:27.721366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:56017)
0927 03:54:27.721499 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:27.721510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:27.721578 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:27.721703 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:27.721704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:27.721872 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:27.721915 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.721916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.722622 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.722623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.723164 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:27.723166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.723505 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.723506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:27.723561 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.723989 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:27.723994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:27.725790 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260927 03:54:27.725957 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:27.721406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:56017 (local address 127.1.50.126:35029)
0927 03:54:27.721487 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:27.721489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:27.721570 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:54:27.721636 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:27.721637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:27.721650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:27.721698 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:27.721983 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.722547 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.723240 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:27.723446 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:27.723500 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:27.725895 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:56017: BlockingRecv error: recv got EOF from 127.1.50.67:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:28.473269 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:28.468730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:41851)
0927 03:54:28.468898 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:28.468911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:28.468982 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:28.469178 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:28.469178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:28.469354 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:28.469406 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.469407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.470126 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.470128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.470753 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.470756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.471129 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.471130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.471189 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.471591 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:28.471595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:28.473214 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
W20260927 03:54:28.473392 10392 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:28.468960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:41851 (local address 127.1.50.126:35029)
0927 03:54:28.469058 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:28.469060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:28.469068 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:28.469091 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:28.469092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:28.469101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:28.469157 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:28.469469 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.469996 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.470849 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.471070 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.471127 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.473322 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:41851: BlockingRecv error: recv got EOF from 127.1.50.65:41851 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:28.605602 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:28.600987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:45493)
0927 03:54:28.601106 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:28.601118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:28.601188 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:28.601366 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:28.601367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:28.601536 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:28.601581 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.601583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.602411 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.602412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.603126 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.603129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.603441 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.603442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.603495 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.603962 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:28.603967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:28.605528 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
W20260927 03:54:28.605686 10392 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:28.601077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:45493 (local address 127.1.50.126:35029)
0927 03:54:28.601128 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:28.601130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:28.601170 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:54:28.601244 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:28.601245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:28.601258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:28.601351 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:28.601651 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.602343 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.603190 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.603382 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.603436 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.605636 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:45493: BlockingRecv error: recv got EOF from 127.1.50.66:45493 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:28 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:28.731123 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:28.726403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:34627)
0927 03:54:28.726553 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:28.726563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:28.726638 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:28.726782 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:28.726782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:28.727035 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:28.727086 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.727089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.727843 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.727844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.728404 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:28.728406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.728765 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.728766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:28.728816 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.729260 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:28.729265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:28.731064 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
W20260927 03:54:28.731213 10392 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:28.726499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:34627 (local address 127.1.50.126:35029)
0927 03:54:28.726564 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:28.726566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:28.726629 (+ 63us) server_negotiation.cc:407] Connection header received
0927 03:54:28.726695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:28.726696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:28.726707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:28.726768 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:28.727192 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.727742 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.728491 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:28.728686 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:28.728740 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:28.731154 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:34627: BlockingRecv error: recv got EOF from 127.1.50.67:34627 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:29.478353 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:29.473788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:60087)
0927 03:54:29.473910 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:29.473921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:29.473974 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:29.474160 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:29.474161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:29.474334 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:29.474382 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.474384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.475065 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.475066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.475600 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.475602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.476002 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.476003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.476043 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.476463 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:29.476468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:29.478296 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
W20260927 03:54:29.478475 10396 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:29.473889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:60087 (local address 127.1.50.126:35029)
0927 03:54:29.474034 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:29.474035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:29.474043 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:29.474065 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:29.474066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:29.474076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:29.474142 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:29.474442 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.474976 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.475727 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.475925 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.475986 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.478408 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:60087: BlockingRecv error: recv got EOF from 127.1.50.65:60087 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:29.614077 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:29.606101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:54477)
0927 03:54:29.606222 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:29.606234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:29.606307 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:29.606535 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:29.606536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:29.606704 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:29.606754 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.606755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.607360 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.607360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.607893 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.607894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.608232 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.608233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.608288 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.612023 (+ 3735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:29.612028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:29.613979 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260927 03:54:29.614212 10396 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:29.606370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54477 (local address 127.1.50.126:35029)
0927 03:54:29.606423 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:29.606425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:29.606432 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:29.606455 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:29.606456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:29.606466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:29.606515 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:29.606814 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.607299 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.607964 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.608151 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.608202 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.614144 (+ 5942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:54477: BlockingRecv error: recv got EOF from 127.1.50.66:54477 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:29 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:29.735952 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:29.731581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:43087)
0927 03:54:29.731735 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:29.731747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:29.731811 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:29.731933 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:29.731933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:29.732126 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:29.732168 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.732171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.732851 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.732852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.733361 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:29.733363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.733707 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.733708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:29.733761 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.734236 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:29.734241 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:29.735873 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
W20260927 03:54:29.736016 10396 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:29.731701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:43087 (local address 127.1.50.126:35029)
0927 03:54:29.731764 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:29.731766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:29.731792 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:54:29.731864 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:29.731865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:29.731878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:29.731924 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:29.732237 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.732776 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.733430 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:29.733639 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:29.733699 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:29.735963 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:43087: BlockingRecv error: recv got EOF from 127.1.50.67:43087 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:30.486579 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:30.478853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:43543)
0927 03:54:30.478983 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:30.478993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:30.479038 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:30.479195 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:30.479195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:30.479347 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:30.479395 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.479397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.480006 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.480007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.480556 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.480558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.480889 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.480890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.480932 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.484896 (+ 3964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:30.484901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:30.486522 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
W20260927 03:54:30.486691 10400 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:30.478931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:43543 (local address 127.1.50.126:35029)
0927 03:54:30.479042 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:30.479044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:30.479052 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:30.479073 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:30.479073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:30.479083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:30.479170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:30.479452 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.479940 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.480626 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.480821 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.480871 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.486626 (+ 5755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:43543: BlockingRecv error: recv got EOF from 127.1.50.65:43543 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:30.620194 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:30.614767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:50443)
0927 03:54:30.614890 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:30.614902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:30.614956 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:30.615229 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:30.615230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:30.615401 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:30.615450 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.615452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.616249 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.616250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.616808 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.616810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.617156 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.617157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.617209 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.617723 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:30.617727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:30.620128 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260927 03:54:30.620328 10400 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:30.614903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:50443 (local address 127.1.50.126:35029)
0927 03:54:30.615007 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:30.615009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:30.615025 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:54:30.615061 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:30.615062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:30.615074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:30.615163 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:30.615590 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.616163 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.616881 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.617088 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.617138 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.620246 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:50443: BlockingRecv error: recv got EOF from 127.1.50.66:50443 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:30 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:30.742784 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:30.736591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:33465)
0927 03:54:30.736734 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:30.736747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:30.736822 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:30.736987 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:30.736989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:30.737188 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:30.737254 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.737257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.738269 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.738270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.739066 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:30.739070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.739485 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.739486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:30.739544 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.740099 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:30.740105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:30.742690 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
W20260927 03:54:30.742904 10400 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:30.736725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:33465 (local address 127.1.50.126:35029)
0927 03:54:30.736795 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:30.736798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:30.736811 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:54:30.736882 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:30.736883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:30.736902 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:30.736947 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:30.737326 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.738184 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.739151 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:30.739422 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:30.739486 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:30.742828 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:33465: BlockingRecv error: recv got EOF from 127.1.50.67:33465 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:31.492803 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:31.487144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:50437)
0927 03:54:31.487291 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:31.487301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:31.487382 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:31.487769 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:31.487771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:31.488066 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:31.488128 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.488131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.488893 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.488894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.489458 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.489461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.489910 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.489913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.490004 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.490674 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:31.490679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:31.492712 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
W20260927 03:54:31.492957 10404 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:31.487311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:50437 (local address 127.1.50.126:35029)
0927 03:54:31.487586 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:31.487589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:31.487600 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:31.487632 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:31.487633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:31.487649 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:31.487738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:31.488201 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.488818 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.489539 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.489758 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.489839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.492862 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:50437: BlockingRecv error: recv got EOF from 127.1.50.65:50437 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":206,"threads_started":1}
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:31.626103 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:31.620773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:60879)
0927 03:54:31.620980 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:31.620995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:31.621074 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:31.621222 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:31.621223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:31.621466 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:31.621525 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.621528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.622337 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.622338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.622937 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.622940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.623295 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.623296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.623353 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.623980 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:31.623985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:31.625972 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
W20260927 03:54:31.626243 10404 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:31.620876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:60879 (local address 127.1.50.126:35029)
0927 03:54:31.620962 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:31.620964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:31.621066 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:54:31.621135 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:31.621136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:31.621151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:31.621207 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:31.621606 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.622253 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.623028 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.623234 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.623290 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.626173 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:60879: BlockingRecv error: recv got EOF from 127.1.50.66:60879 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:31 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:31.748905 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:31.743441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:49675)
0927 03:54:31.743647 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:31.743660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:31.743736 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:31.743931 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:31.743932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:31.744156 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:31.744212 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.744214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.744977 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.744977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.745540 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:31.745542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.745926 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.745927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:31.746010 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.746530 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:31.746535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:31.748817 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
W20260927 03:54:31.749039 10404 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:31.743562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:49675 (local address 127.1.50.126:35029)
0927 03:54:31.743652 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:31.743655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:31.743727 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:54:31.743825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:31.743826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:31.743845 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:31.743907 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:31.744290 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.744884 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.745628 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:31.745856 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:31.745919 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:31.748943 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:49675: BlockingRecv error: recv got EOF from 127.1.50.67:49675 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:32.498698 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:32.493339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:49719)
0927 03:54:32.493493 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:32.493508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:32.493572 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:32.493920 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:32.493921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:32.494170 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:32.494239 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.494241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.494968 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.494969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.495571 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.495573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.495974 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.495975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.496033 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.496573 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:32.496578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:32.498614 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260927 03:54:32.498858 10408 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:32.493527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:49719 (local address 127.1.50.126:35029)
0927 03:54:32.493686 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:32.493688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:32.493700 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:54:32.493730 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:32.493731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:32.493743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:32.493864 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:32.494321 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.494880 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.495648 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.495885 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.495945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.498736 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:49719: BlockingRecv error: recv got EOF from 127.1.50.65:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:32.631906 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:32.626683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:56259)
0927 03:54:32.626897 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:32.626913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:32.627008 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:32.627186 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:32.627187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:32.627401 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:32.627464 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.627466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.628187 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.628188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.628778 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.628780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.629126 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.629127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.629184 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.629703 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:32.629707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:32.631827 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":133,"threads_started":1}
W20260927 03:54:32.632033 10408 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:32.626817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:56259 (local address 127.1.50.126:35029)
0927 03:54:32.626903 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:32.626906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:32.626990 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:54:32.627106 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:32.627107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:32.627123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:32.627173 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:32.627536 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.628111 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.628858 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.629066 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.629124 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.631943 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:56259: BlockingRecv error: recv got EOF from 127.1.50.66:56259 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:32 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:32.754096 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:32.749433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:45701)
0927 03:54:32.749589 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:32.749601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:32.749667 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:32.749860 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:32.749861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:32.750047 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:32.750093 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.750095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.750776 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.750777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.751284 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:32.751286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.751601 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.751602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:32.751654 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.752106 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:32.752110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:32.753979 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
W20260927 03:54:32.754179 10408 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:32.749570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:45701 (local address 127.1.50.126:35029)
0927 03:54:32.749654 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:32.749656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:32.749668 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:54:32.749737 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:32.749739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:32.749754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:32.749802 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:32.750176 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.750695 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.751362 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:32.751543 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:32.751597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:32.754119 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:45701: BlockingRecv error: recv got EOF from 127.1.50.67:45701 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:33.503806 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:33.499220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:46371)
0927 03:54:33.499356 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:33.499371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:33.499444 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:33.499592 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:33.499593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:33.499769 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:33.499826 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.499828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.500490 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.500491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.501041 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.501043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.501464 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.501466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.501515 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.501959 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:33.501964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:33.503741 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
W20260927 03:54:33.503932 10412 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:33.499329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:46371 (local address 127.1.50.126:35029)
0927 03:54:33.499454 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:33.499456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:33.499466 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:33.499508 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:33.499508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:33.499518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:33.499575 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:33.499893 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.500412 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.501141 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.501378 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.501443 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.503858 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:46371: BlockingRecv error: recv got EOF from 127.1.50.65:46371 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:33.636804 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:33.632398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:54567)
0927 03:54:33.632524 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:33.632537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:33.632594 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:33.632726 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:33.632727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:33.632905 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:33.632964 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.632967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.633626 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.633627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.634196 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.634199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.634533 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.634534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.634577 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.635067 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:33.635072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:33.636733 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
W20260927 03:54:33.637007 10412 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:33.632510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:54567 (local address 127.1.50.126:35029)
0927 03:54:33.632565 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:33.632567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:33.632592 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:54:33.632651 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:33.632652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:33.632664 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:33.632709 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:33.633037 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.633539 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.634279 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.634469 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.634524 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.636961 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:54567: BlockingRecv error: recv got EOF from 127.1.50.66:54567 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:33 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:33.759223 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:33.754696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:49019)
0927 03:54:33.754816 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:33.754826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:33.754884 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:33.755004 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:33.755005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:33.755161 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:33.755205 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.755207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.755872 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.755874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.756411 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:33.756413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.756813 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.756814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:33.756865 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.757295 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:33.757300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:33.759146 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
W20260927 03:54:33.759303 10412 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:33.754809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:49019 (local address 127.1.50.126:35029)
0927 03:54:33.754867 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:33.754869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:33.754879 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:33.754927 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:33.754928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:33.754941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:33.754987 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:33.755262 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.755769 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.756494 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:33.756678 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:33.756732 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:33.759249 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:49019: BlockingRecv error: recv got EOF from 127.1.50.67:49019 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:34.508793 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:34.504339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:55027)
0927 03:54:34.504453 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:34.504463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:34.504523 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:34.504638 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:34.504638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:34.504802 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:34.504850 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.504852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.505555 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.505556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.506143 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.506145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.506533 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.506534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.506593 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.507088 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:34.507093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:34.508723 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260927 03:54:34.508934 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:34.504402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55027 (local address 127.1.50.126:35029)
0927 03:54:34.504500 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:34.504501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:34.504511 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:34.504567 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:34.504568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:34.504578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:34.504633 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:34.504948 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.505480 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.506229 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.506448 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.506524 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.508857 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:55027: BlockingRecv error: recv got EOF from 127.1.50.65:55027 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:34.647017 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:34.637317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:38457)
0927 03:54:34.637435 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:34.637446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:34.637495 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:34.637637 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:34.637637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:34.637796 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:34.637840 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.637842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.638496 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.638497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.639057 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.639059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.639448 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.639449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.639502 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.644921 (+ 5419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:34.644926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:34.646954 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
W20260927 03:54:34.647147 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:34.637420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:38457 (local address 127.1.50.126:35029)
0927 03:54:34.637473 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:34.637475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:34.637485 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:54:34.637552 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:34.637552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:34.637562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:34.637619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:34.637896 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.638415 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.639155 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.639357 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.639425 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.647088 (+ 7663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:38457: BlockingRecv error: recv got EOF from 127.1.50.66:38457 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:34 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.67@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:34.764381 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:34.759861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.67:44623)
0927 03:54:34.759987 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:34.759997 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:34.760056 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:34.760191 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:34.760191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:34.760371 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:34.760416 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.760418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.761103 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.761104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.761698 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:34.761701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.762140 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.762141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:34.762181 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.762633 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:34.762637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:34.764296 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260927 03:54:34.764457 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:34.759923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:44623 (local address 127.1.50.126:35029)
0927 03:54:34.760004 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:34.760006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:34.760045 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:54:34.760121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:34.760122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:34.760134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:34.760180 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:34.760490 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.761032 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.761798 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:34.761987 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:34.762063 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:34.764395 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.67:44623: BlockingRecv error: recv got EOF from 127.1.50.67:44623 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 03:54:35 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:35 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.65@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:35.513790 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:35.509431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.65:45015)
0927 03:54:35.509580 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:35.509591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:35.509652 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:35.509802 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:35.509802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:35.509961 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:35.510006 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.510007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.510704 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.510705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.511208 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.511209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.511552 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.511553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.511604 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.512082 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:35.512088 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:35.513734 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260927 03:54:35.513921 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:35.509563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:45015 (local address 127.1.50.126:35029)
0927 03:54:35.509687 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:35.509689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:35.509697 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:35.509722 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:35.509723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:35.509733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:35.509788 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:35.510120 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.510625 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.511278 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.511472 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.511551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.513843 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.65:45015: BlockingRecv error: recv got EOF from 127.1.50.65:45015 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":50,"threads_started":1}
Sep 27 03:54:35 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:54:35 dist-test-slave-96fg krb5kdc[9362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.50.66@KRBTEST.COM for oryx/127.1.50.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:54:35.652195 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:35.647692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:35029 (local address 127.1.50.66:46041)
0927 03:54:35.647820 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:35.647832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:35.647896 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:35.648065 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:35.648066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:35.648228 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:35.648278 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.648279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.648966 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.648967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.649515 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.649517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.649833 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.649834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.649876 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.650332 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:35.650337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:35.652127 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.50.126:35029: Server oryx/127.1.50.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260927 03:54:35.652285 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:35.647732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:46041 (local address 127.1.50.126:35029)
0927 03:54:35.647836 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:35.647837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:35.647889 (+ 52us) server_negotiation.cc:407] Connection header received
0927 03:54:35.647962 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:35.647963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:35.647973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:35.648018 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:35.648361 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.648881 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.649583 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.649760 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.649827 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.652225 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.50.66:46041: BlockingRecv error: recv got EOF from 127.1.50.66:46041 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:54:35.675709 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9945
I20260927 03:54:35.679785 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10097
I20260927 03:54:35.683540 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10229
I20260927 03:54:35.687160 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 9875
2026-09-27T03:54:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17427 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T03:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:35Z Disabled control of system clock
I20260927 03:54:35.714186 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46189
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:41991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:46189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:54:35.792222 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:35.792423 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:35.792443 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:35.793542 10425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:35.793574 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:35.793584 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:35.793596 10425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:35.793605 10425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:35.794904 10425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:46189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:35.795096 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:35.795337 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:35.800190 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:35.800149 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:35.800370 10425 server_base.cc:1034] running on GCE node
W20260927 03:54:35.800153 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:35.800666 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:35.800966 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:35.802073 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790481275802054 us; error 38 us; skew 500 ppm
I20260927 03:54:35.803210 10425 webserver.cc:460] Webserver started at http://127.1.50.126:37673/ using document root <none> and password file <none>
I20260927 03:54:35.803447 10425 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:35.803493 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:35.803618 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:35.804703 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "615a0d0c4d6d43939313dec552b33608"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "db476ac83320e3726585744638a8f9cf"
server_key_iv: "34b2fff8c65fc92f0ef80e620b027a5e"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.805029 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "615a0d0c4d6d43939313dec552b33608"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "db476ac83320e3726585744638a8f9cf"
server_key_iv: "34b2fff8c65fc92f0ef80e620b027a5e"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.806293 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:54:35.806978 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:35.807106 10425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:35.807192 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "615a0d0c4d6d43939313dec552b33608"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "db476ac83320e3726585744638a8f9cf"
server_key_iv: "34b2fff8c65fc92f0ef80e620b027a5e"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.807246 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:35.835110 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:35.835427 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:35.835552 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:35.839318 10425 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:46189
I20260927 03:54:35.839402 10491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:46189 every 8 connection(s)
I20260927 03:54:35.839675 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:35.840314 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:35.842911 10492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608: Bootstrap starting.
I20260927 03:54:35.843442 10492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:35.843675 10492 log.cc:824] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:35.844261 10492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608: No bootstrap required, opened a new log
I20260927 03:54:35.845770 10492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } }
I20260927 03:54:35.845911 10492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:35.845933 10492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615a0d0c4d6d43939313dec552b33608, State: Initialized, Role: FOLLOWER
I20260927 03:54:35.846073 10492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } }
I20260927 03:54:35.846155 10492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:35.846185 10492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:35.846218 10492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:35.846755 10492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } }
I20260927 03:54:35.846904 10492 leader_election.cc:302] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615a0d0c4d6d43939313dec552b33608; no voters:
I20260927 03:54:35.847091 10492 leader_election.cc:288] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:35.847149 10495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:35.847297 10495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [term 1 LEADER]: Becoming Leader. State: Replica: 615a0d0c4d6d43939313dec552b33608, State: Running, Role: LEADER
I20260927 03:54:35.847385 10492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:35.847412 10495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } }
I20260927 03:54:35.847908 10497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615a0d0c4d6d43939313dec552b33608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } } }
I20260927 03:54:35.847990 10497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:35.848212 10496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615a0d0c4d6d43939313dec552b33608. Latest consensus state: current_term: 1 leader_uuid: "615a0d0c4d6d43939313dec552b33608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a0d0c4d6d43939313dec552b33608" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46189 } } }
I20260927 03:54:35.848278 10496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:35.848322 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10425
I20260927 03:54:35.848385 10504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:35.848404 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:35.848584 1225 external_mini_cluster.cc:1428] Setting key f16d40e2190ac9584faf5e6c1282d3e5
I20260927 03:54:35.848898 10504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:35.850415 10504 catalog_manager.cc:1347] Generated new cluster ID: 158c1ff763c7483d93d2929c1ef7e4eb
I20260927 03:54:35.850464 10504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:35.854858 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:35.849222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.1.50.126:46189)
0927 03:54:35.849676 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:35.849685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:35.849714 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:54:35.849835 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:35.849841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:35.849915 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:35.850268 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:35.850358 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.851662 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.852747 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.853363 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.853478 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.854148 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:35.854153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:35.854153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:35.854376 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:35.854417 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:35.854432 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:35.854497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:35.854562 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:54:35.854639 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":66,"threads_started":1}
I20260927 03:54:35.858508 10504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:35.859093 10504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:35.867337 10504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615a0d0c4d6d43939313dec552b33608: Generated new TSK 0
I20260927 03:54:35.867527 10504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:35.871258 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--builtin_ntp_servers=127.1.50.84:41991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:54:35.951344 10516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:35.951546 10516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:35.951566 10516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:35.952741 10516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:35.952782 10516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:35.952805 10516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:35.954164 10516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:35.954399 10516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:35.954675 10516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:35.961056 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:35.961066 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:35.961071 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:35.961328 10516 server_base.cc:1034] running on GCE node
I20260927 03:54:35.961539 10516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:35.961804 10516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:35.962976 10516 hybrid_clock.cc:648] HybridClock initialized: now 1790481275962960 us; error 30 us; skew 500 ppm
I20260927 03:54:35.964296 10516 webserver.cc:460] Webserver started at http://127.1.50.65:46215/ using document root <none> and password file <none>
I20260927 03:54:35.964529 10516 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:35.964579 10516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:35.964706 10516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:35.965834 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "e31163ebfec3496fad375dcd96d6ebd0"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "089b666e26f0eb35bb8df1823e846c56"
server_key_iv: "75d5df925e7ff0c0292a0b3924dbfe0d"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.966216 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "e31163ebfec3496fad375dcd96d6ebd0"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "089b666e26f0eb35bb8df1823e846c56"
server_key_iv: "75d5df925e7ff0c0292a0b3924dbfe0d"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.967586 10516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:35.968433 10530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:35.968598 10516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:35.968686 10516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "e31163ebfec3496fad375dcd96d6ebd0"
format_stamp: "Formatted at 2026-09-27 03:54:35 on dist-test-slave-96fg"
server_key: "089b666e26f0eb35bb8df1823e846c56"
server_key_iv: "75d5df925e7ff0c0292a0b3924dbfe0d"
server_key_version: "encryptionkey@0"
I20260927 03:54:35.968757 10516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:35.983273 10516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:35.983579 10516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:35.983767 10516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:35.983989 10516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:35.984308 10516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:35.984341 10516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:35.984370 10516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:35.984392 10516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:35.989702 10516 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:33419
I20260927 03:54:35.989761 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:33419 every 8 connection(s)
I20260927 03:54:35.990069 10516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:35.995193 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:35.990752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44291 (local address 127.1.50.126:46189)
0927 03:54:35.990800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:35.990803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:35.991021 (+ 218us) server_negotiation.cc:407] Connection header received
0927 03:54:35.991475 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:35.991476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:35.991488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:35.991535 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:35.992156 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.992646 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.993642 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.993899 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.993957 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.994818 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:35.994820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:35.994820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:35.994827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:35.994854 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:35.994858 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:35.994979 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:35.995089 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:54:35.995125 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:54:35.995393 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:35.990735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46189 (local address 127.1.50.65:44291)
0927 03:54:35.990910 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:35.990942 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:35.991404 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:35.991586 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:35.991589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:35.991598 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:35.992063 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.992067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.992756 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.992757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.993543 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:35.993546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:35.993980 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:35.993981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:35.994155 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:35.994641 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:35.994642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:35.994671 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:35.994695 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:35.994999 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:35.995005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:35.995008 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:35.995079 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:54:35.995168 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 03:54:35.995837 10644 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46189
I20260927 03:54:35.995916 10644 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:35.996085 10644 heartbeater.cc:511] Master 127.1.50.126:46189 requested a full tablet report, sending...
I20260927 03:54:35.996624 10456 ts_manager.cc:194] Registered new tserver with Master: e31163ebfec3496fad375dcd96d6ebd0 (127.1.50.65:33419)
I20260927 03:54:35.996906 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10516
I20260927 03:54:35.996984 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:35.997143 1225 external_mini_cluster.cc:1428] Setting key 22b14c440cdac11f91a7dba814ae467c
I20260927 03:54:35.997346 10456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.65:44291
I20260927 03:54:35.998170 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--builtin_ntp_servers=127.1.50.84:41991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:54:36.077208 10647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.077441 10647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.077469 10647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.078646 10647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.078683 10647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.078716 10647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:36.080003 10647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.080229 10647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.080503 10647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:36.085173 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.085649 10653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.085700 10647 server_base.cc:1034] running on GCE node
W20260927 03:54:36.085649 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.086004 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.086349 10647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.087508 10647 hybrid_clock.cc:648] HybridClock initialized: now 1790481276087493 us; error 30 us; skew 500 ppm
I20260927 03:54:36.088899 10647 webserver.cc:460] Webserver started at http://127.1.50.66:44705/ using document root <none> and password file <none>
I20260927 03:54:36.089148 10647 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.089202 10647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.089332 10647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.090482 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "1ca64ba1c4a843e18eedbfe19b667984"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "90068eadc7ea7a52838662f58eb598a4"
server_key_iv: "bbc10cc3bf4346833ca620ac59b41156"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.090845 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "1ca64ba1c4a843e18eedbfe19b667984"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "90068eadc7ea7a52838662f58eb598a4"
server_key_iv: "bbc10cc3bf4346833ca620ac59b41156"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.092131 10647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:36.092823 10661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.092976 10647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:36.093052 10647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "1ca64ba1c4a843e18eedbfe19b667984"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "90068eadc7ea7a52838662f58eb598a4"
server_key_iv: "bbc10cc3bf4346833ca620ac59b41156"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.093111 10647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.112012 10647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.112339 10647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.112468 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.112690 10647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:36.113013 10647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:36.113044 10647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.113073 10647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:36.113089 10647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.118448 10647 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:42913
I20260927 03:54:36.118502 10774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:42913 every 8 connection(s)
I20260927 03:54:36.118803 10647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:36.122817 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10647
I20260927 03:54:36.122912 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:36.123081 1225 external_mini_cluster.cc:1428] Setting key ba2ca487edc05078a9ac48dfa49fb28e
I20260927 03:54:36.123907 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--builtin_ntp_servers=127.1.50.84:41991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:54:36.124653 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.119612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:60059 (local address 127.1.50.126:46189)
0927 03:54:36.119687 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.119690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.119941 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:54:36.120435 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.120436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.120446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.120497 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.121177 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.121704 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.122899 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.123195 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.123272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:36.124128 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.124129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.124129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.124134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.124149 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.124219 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.124294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.124540 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.124571 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:54:36.124863 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.119577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46189 (local address 127.1.50.66:60059)
0927 03:54:36.119820 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:36.119857 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:36.120351 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:36.120553 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:36.120557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:36.120565 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:36.121062 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:36.121065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.121828 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.121829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:36.122730 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:36.122734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.123305 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.123307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:36.123434 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:36.123969 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:36.123969 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:36.124012 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:36.124047 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:36.124309 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:36.124446 (+ 137us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:36.124451 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:36.124532 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:54:36.124651 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260927 03:54:36.125306 10775 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46189
I20260927 03:54:36.125425 10775 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:36.125614 10775 heartbeater.cc:511] Master 127.1.50.126:46189 requested a full tablet report, sending...
I20260927 03:54:36.126122 10455 ts_manager.cc:194] Registered new tserver with Master: 1ca64ba1c4a843e18eedbfe19b667984 (127.1.50.66:42913)
I20260927 03:54:36.126492 10455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.66:60059
W20260927 03:54:36.202677 10778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.202898 10778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.202927 10778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.204120 10778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.204157 10778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.204190 10778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:36.205607 10778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.205858 10778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.206156 10778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:36.210970 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.210970 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.210980 10783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.211383 10778 server_base.cc:1034] running on GCE node
I20260927 03:54:36.211586 10778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.211810 10778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.212954 10778 hybrid_clock.cc:648] HybridClock initialized: now 1790481276212943 us; error 34 us; skew 500 ppm
I20260927 03:54:36.214339 10778 webserver.cc:460] Webserver started at http://127.1.50.67:35321/ using document root <none> and password file <none>
I20260927 03:54:36.214576 10778 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.214622 10778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.214751 10778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.215863 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "b31e0642ceaa48cc9bdee396cc27e7b5"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "ddf3e840258ff263917aa8a08b09ae12"
server_key_iv: "f9f51c8c1cc87efe29b8143efbf685a5"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.216182 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "b31e0642ceaa48cc9bdee396cc27e7b5"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "ddf3e840258ff263917aa8a08b09ae12"
server_key_iv: "f9f51c8c1cc87efe29b8143efbf685a5"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.217440 10778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:36.218281 10792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.218472 10778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:36.218555 10778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "b31e0642ceaa48cc9bdee396cc27e7b5"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "ddf3e840258ff263917aa8a08b09ae12"
server_key_iv: "f9f51c8c1cc87efe29b8143efbf685a5"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.218631 10778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.235456 10778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.235759 10778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.235891 10778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.236094 10778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:36.236400 10778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:36.236433 10778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.236461 10778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:36.236483 10778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.241732 10778 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:46009
I20260927 03:54:36.241788 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:46009 every 8 connection(s)
I20260927 03:54:36.242138 10778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:36.246989 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.242805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:53073 (local address 127.1.50.126:46189)
0927 03:54:36.242906 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.242908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.243070 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:54:36.243556 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.243558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.243569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.243631 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.244272 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.244767 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.245691 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.245881 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.245935 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:36.246723 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.246724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.246724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.246729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.246742 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.246745 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.246784 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.246898 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.246927 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 03:54:36.247174 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.242776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46189 (local address 127.1.50.67:53073)
0927 03:54:36.242957 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:36.242996 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:36.243452 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:36.243698 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:36.243702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:36.243710 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:36.244193 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:36.244197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.244855 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.244856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:36.245618 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:36.245620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:36.245973 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:36.245974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:36.246111 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:36.246600 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:36.246600 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:36.246629 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:36.246651 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:36.246790 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:36.246816 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:36.246818 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:54:36.246868 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:54:36.246974 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260927 03:54:36.247555 10906 heartbeater.cc:347] Connected to a master server at 127.1.50.126:46189
I20260927 03:54:36.247638 10906 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:36.247825 10906 heartbeater.cc:511] Master 127.1.50.126:46189 requested a full tablet report, sending...
I20260927 03:54:36.248164 10455 ts_manager.cc:194] Registered new tserver with Master: b31e0642ceaa48cc9bdee396cc27e7b5 (127.1.50.67:46009)
I20260927 03:54:36.248514 10455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.67:53073
I20260927 03:54:36.249096 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10778
I20260927 03:54:36.249158 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:36.249300 1225 external_mini_cluster.cc:1428] Setting key f7d9c26a0fa5d849bb50828aa1238438
I20260927 03:54:36.250005 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:54:36.251626 10511 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:36.251205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.1.50.126:46189)
0927 03:54:36.251244 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.251246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.251374 (+ 128us) server_negotiation.cc:407] Connection header received
0927 03:54:36.251396 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.251397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.251406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.251447 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.251565 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:54:36.251698 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10516
I20260927 03:54:36.256392 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10647
I20260927 03:54:36.259919 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10778
I20260927 03:54:36.263413 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10425
2026-09-27T03:54:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (574 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T03:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:36Z Disabled control of system clock
I20260927 03:54:36.288039 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36475
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:36475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:54:36.367218 10917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.367431 10917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.367451 10917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.368629 10917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:36.368669 10917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.368681 10917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.368692 10917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:36.368703 10917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:36.370121 10917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:36475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.370368 10917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.370645 10917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:36.375229 10925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.375208 10923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.375253 10917 server_base.cc:1034] running on GCE node
W20260927 03:54:36.375226 10922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.375612 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.375931 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.377079 10917 hybrid_clock.cc:648] HybridClock initialized: now 1790481276377070 us; error 28 us; skew 500 ppm
I20260927 03:54:36.378202 10917 webserver.cc:460] Webserver started at http://127.1.50.126:37773/ using document root <none> and password file <none>
I20260927 03:54:36.378412 10917 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.378465 10917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.378583 10917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.379604 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "6d839c997c3048609e5529107fe72fca"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "0f39190317b34ee9f1b720fd6d448f57"
server_key_iv: "8d83e77d72bf7f8bed1ca4bda9100516"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.379923 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "6d839c997c3048609e5529107fe72fca"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "0f39190317b34ee9f1b720fd6d448f57"
server_key_iv: "8d83e77d72bf7f8bed1ca4bda9100516"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.381143 10917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:36.381850 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.381974 10917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:36.382073 10917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "6d839c997c3048609e5529107fe72fca"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "0f39190317b34ee9f1b720fd6d448f57"
server_key_iv: "8d83e77d72bf7f8bed1ca4bda9100516"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.382141 10917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.385582 10917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.385842 10917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.385957 10917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.389307 10917 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:36475
I20260927 03:54:36.389359 10983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:36475 every 8 connection(s)
I20260927 03:54:36.389640 10917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:36.390312 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:36.391855 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 10917
I20260927 03:54:36.391927 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:36.392083 1225 external_mini_cluster.cc:1428] Setting key 251333293d9964c3db9d0ad7476ea57d
I20260927 03:54:36.393436 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca: Bootstrap starting.
I20260927 03:54:36.394122 10984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:36.394426 10984 log.cc:824] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:36.394490 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.392593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.1.50.126:36475)
0927 03:54:36.392796 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.392804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.392835 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:54:36.393297 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.393305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.393363 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.393513 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.394063 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.394066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.394066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.394071 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.394123 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.394138 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.394225 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.394260 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.394318 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260927 03:54:36.395093 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca: No bootstrap required, opened a new log
I20260927 03:54:36.396312 10984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } }
I20260927 03:54:36.396433 10984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:36.396453 10984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d839c997c3048609e5529107fe72fca, State: Initialized, Role: FOLLOWER
I20260927 03:54:36.396539 10984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } }
I20260927 03:54:36.396626 10984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:36.396652 10984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:36.396683 10984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:36.397357 10984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } }
I20260927 03:54:36.397439 10984 leader_election.cc:302] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d839c997c3048609e5529107fe72fca; no voters:
I20260927 03:54:36.397544 10984 leader_election.cc:288] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:36.397598 10989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:36.397737 10989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [term 1 LEADER]: Becoming Leader. State: Replica: 6d839c997c3048609e5529107fe72fca, State: Running, Role: LEADER
I20260927 03:54:36.397771 10984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:36.397826 10989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } }
I20260927 03:54:36.398154 10991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d839c997c3048609e5529107fe72fca. Latest consensus state: current_term: 1 leader_uuid: "6d839c997c3048609e5529107fe72fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } } }
I20260927 03:54:36.398243 10991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:36.398144 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d839c997c3048609e5529107fe72fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d839c997c3048609e5529107fe72fca" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36475 } } }
I20260927 03:54:36.398397 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:36.398502 10996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:36.398950 10996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:36.400295 10996 catalog_manager.cc:1347] Generated new cluster ID: ce0c9c710b74430a8cadb0ac444e4e36
I20260927 03:54:36.400382 10996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:36.417661 10996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:36.417831 10996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:36.420038 10996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d839c997c3048609e5529107fe72fca: Generated new TSK 0
I20260927 03:54:36.420202 10996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:54:36.424587 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--builtin_ntp_servers=127.1.50.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:54:36.521405 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.521605 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.521627 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.522825 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.522862 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.522887 11008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:36.524186 11008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.524400 11008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.524669 11008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:54:36.529418 11008 server_base.cc:1034] running on GCE node
W20260927 03:54:36.529378 11013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.529389 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.529510 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.529814 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.530085 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.531237 11008 hybrid_clock.cc:648] HybridClock initialized: now 1790481276531221 us; error 31 us; skew 500 ppm
I20260927 03:54:36.532429 11008 webserver.cc:460] Webserver started at http://127.1.50.65:33595/ using document root <none> and password file <none>
I20260927 03:54:36.532639 11008 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.532691 11008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.532810 11008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.533845 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "7cfb4411494d41d3869912aec7cdba2e"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "cf722abe3fc040822f6d53e7f8a1dfe0"
server_key_iv: "03aa54592f2043b09ecb0b689af66c7d"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.534194 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "7cfb4411494d41d3869912aec7cdba2e"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "cf722abe3fc040822f6d53e7f8a1dfe0"
server_key_iv: "03aa54592f2043b09ecb0b689af66c7d"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.535462 11008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:36.536198 11022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.536358 11008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:36.536437 11008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "7cfb4411494d41d3869912aec7cdba2e"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "cf722abe3fc040822f6d53e7f8a1dfe0"
server_key_iv: "03aa54592f2043b09ecb0b689af66c7d"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.536499 11008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.539708 11008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.539955 11008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.540073 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.540269 11008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:36.540550 11008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:36.540582 11008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.540608 11008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:36.540632 11008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.545825 11008 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:39709
I20260927 03:54:36.545892 11135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:39709 every 8 connection(s)
I20260927 03:54:36.546216 11008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:36.549199 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.546991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:59267 (local address 127.1.50.126:36475)
0927 03:54:36.547033 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.547035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.547336 (+ 301us) server_negotiation.cc:407] Connection header received
0927 03:54:36.547805 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.547806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.547813 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.547874 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.548977 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.548979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.548979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.548984 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.548999 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.549003 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.549052 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.549119 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.549132 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:54:36.549396 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.547010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36475 (local address 127.1.50.65:59267)
0927 03:54:36.547207 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:36.547249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:36.547712 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:36.547953 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:36.547957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:36.547967 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:36.548806 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:36.548807 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:36.548867 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:36.548894 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:36.549056 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:36.549060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:36.549063 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:36.549129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:54:36.549200 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260927 03:54:36.549631 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11008
I20260927 03:54:36.549710 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:36.549902 1225 external_mini_cluster.cc:1428] Setting key e558009415ea6aa8054779cdd28bf5ca
I20260927 03:54:36.549913 11136 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36475
I20260927 03:54:36.549989 11136 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:36.550146 11136 heartbeater.cc:511] Master 127.1.50.126:36475 requested a full tablet report, sending...
I20260927 03:54:36.550616 10947 ts_manager.cc:194] Registered new tserver with Master: 7cfb4411494d41d3869912aec7cdba2e (127.1.50.65:39709)
I20260927 03:54:36.550848 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--builtin_ntp_servers=127.1.50.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:54:36.629004 11139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.629213 11139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.629235 11139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.630384 11139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.630422 11139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.630446 11139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:36.631714 11139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.631888 11139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.632129 11139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:36.637019 11145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.636992 11147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.637095 11139 server_base.cc:1034] running on GCE node
W20260927 03:54:36.637007 11144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.637444 11139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.637678 11139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.638844 11139 hybrid_clock.cc:648] HybridClock initialized: now 1790481276638828 us; error 26 us; skew 500 ppm
I20260927 03:54:36.640035 11139 webserver.cc:460] Webserver started at http://127.1.50.66:36271/ using document root <none> and password file <none>
I20260927 03:54:36.640236 11139 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.640276 11139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.640367 11139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.641404 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "5523b9193ccc48578088550954f3f35d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "99ff46b378422da83e2cfe9d6e4ae9e3"
server_key_iv: "d357d4d0f6cf8b90af2cf26c5f9b4292"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.641693 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "5523b9193ccc48578088550954f3f35d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "99ff46b378422da83e2cfe9d6e4ae9e3"
server_key_iv: "d357d4d0f6cf8b90af2cf26c5f9b4292"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.642968 11139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:36.643702 11153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.643821 11139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:54:36.643908 11139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "5523b9193ccc48578088550954f3f35d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "99ff46b378422da83e2cfe9d6e4ae9e3"
server_key_iv: "d357d4d0f6cf8b90af2cf26c5f9b4292"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.643973 11139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.647310 11139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.647547 11139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.647656 11139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.647831 11139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:36.648101 11139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:36.648132 11139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.648154 11139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:36.648175 11139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.653479 11139 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:45003
I20260927 03:54:36.653538 11266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:45003 every 8 connection(s)
I20260927 03:54:36.653827 11139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:36.655099 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11139
I20260927 03:54:36.655176 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:36.655329 1225 external_mini_cluster.cc:1428] Setting key b3d56c99526807821406d4b74460c3c9
I20260927 03:54:36.656244 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--builtin_ntp_servers=127.1.50.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 03:54:36.656733 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.654534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:42449 (local address 127.1.50.126:36475)
0927 03:54:36.654611 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.654614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.654804 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:54:36.655287 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.655288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.655296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.655336 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.656499 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.656500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.656500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.656505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.656521 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.656525 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.656574 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.656649 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.656663 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:54:36.656917 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.654514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36475 (local address 127.1.50.66:42449)
0927 03:54:36.654683 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:36.654716 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:36.655184 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:36.655417 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:36.655421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:36.655432 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:36.656347 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:36.656347 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:36.656386 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:36.656421 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:36.656580 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:36.656585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:36.656588 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:36.656655 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:54:36.656716 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 03:54:36.657738 11267 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36475
I20260927 03:54:36.657823 11267 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:36.657955 11267 heartbeater.cc:511] Master 127.1.50.126:36475 requested a full tablet report, sending...
I20260927 03:54:36.658336 10947 ts_manager.cc:194] Registered new tserver with Master: 5523b9193ccc48578088550954f3f35d (127.1.50.66:45003)
W20260927 03:54:36.734858 11270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:36.735060 11270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:36.735081 11270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:36.736240 11270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:36.736281 11270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:36.736305 11270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:36.737607 11270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:36.737802 11270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:36.738065 11270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:36.743222 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.743201 11278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:36.743254 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:36.743486 11270 server_base.cc:1034] running on GCE node
I20260927 03:54:36.743682 11270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:36.743954 11270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:36.745095 11270 hybrid_clock.cc:648] HybridClock initialized: now 1790481276745079 us; error 36 us; skew 500 ppm
I20260927 03:54:36.746366 11270 webserver.cc:460] Webserver started at http://127.1.50.67:40889/ using document root <none> and password file <none>
I20260927 03:54:36.746601 11270 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:36.746649 11270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:36.746824 11270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:36.747920 11270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "f0dd178d53434dc6bf8b9842401cb67d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "4c2d680b4a01880df5e4e47cd099a6c1"
server_key_iv: "620793fef79b2a0ba4b8becb15147134"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.748229 11270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "f0dd178d53434dc6bf8b9842401cb67d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "4c2d680b4a01880df5e4e47cd099a6c1"
server_key_iv: "620793fef79b2a0ba4b8becb15147134"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.749461 11270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:36.750283 11284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.750427 11270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:36.750509 11270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "f0dd178d53434dc6bf8b9842401cb67d"
format_stamp: "Formatted at 2026-09-27 03:54:36 on dist-test-slave-96fg"
server_key: "4c2d680b4a01880df5e4e47cd099a6c1"
server_key_iv: "620793fef79b2a0ba4b8becb15147134"
server_key_version: "encryptionkey@0"
I20260927 03:54:36.750571 11270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:36.753793 11270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:36.754060 11270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:36.754165 11270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:36.754320 11270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:36.754592 11270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:36.754628 11270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.754647 11270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:36.754664 11270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:36.759886 11270 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:38205
I20260927 03:54:36.759938 11397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:38205 every 8 connection(s)
I20260927 03:54:36.760223 11270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:54:36.760888 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11270
I20260927 03:54:36.760972 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:36.761163 1225 external_mini_cluster.cc:1428] Setting key 66074221602ba227dfcece56fab38ceb
I20260927 03:54:36.763222 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.760946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:41841 (local address 127.1.50.126:36475)
0927 03:54:36.760997 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.761000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.761256 (+ 256us) server_negotiation.cc:407] Connection header received
0927 03:54:36.761884 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.761884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.761891 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.761949 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.763059 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:36.763059 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:36.763060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:54:36.763062 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:36.763075 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:54:36.763078 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:36.763118 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:36.763162 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 03:54:36.763172 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:54:36.763407 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:36.760938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36475 (local address 127.1.50.67:41841)
0927 03:54:36.761119 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:36.761162 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:36.761756 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:36.762048 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:36.762052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:36.762063 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:36.762939 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:54:36.762940 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:36.762968 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:54:36.762994 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:36.763118 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:36.763120 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:36.763122 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:54:36.763167 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:54:36.763213 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260927 03:54:36.763994 11398 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36475
I20260927 03:54:36.764081 11398 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:36.764226 11398 heartbeater.cc:511] Master 127.1.50.126:36475 requested a full tablet report, sending...
I20260927 03:54:36.764534 10947 ts_manager.cc:194] Registered new tserver with Master: f0dd178d53434dc6bf8b9842401cb67d (127.1.50.67:38205)
I20260927 03:54:36.772789 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:54:36.775317 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:54:36.774283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.1.50.126:36475)
0927 03:54:36.774372 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:36.774374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:36.774943 (+ 569us) server_negotiation.cc:407] Connection header received
0927 03:54:36.775006 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:36.775006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:36.775012 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:36.775065 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:36.775246 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:54:36.775521 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11008
I20260927 03:54:36.780078 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11139
I20260927 03:54:36.783444 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11270
I20260927 03:54:36.787428 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 10917
2026-09-27T03:54:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:36 dist-test-slave-96fg krb5kdc[11408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:36 dist-test-slave-96fg krb5kdc[11408](info): set up 2 sockets
Sep 27 03:54:36 dist-test-slave-96fg krb5kdc[11408](info): commencing operation
krb5kdc: starting...
W20260927 03:54:37.738889 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:37 dist-test-slave-96fg krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:54:37.853821 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43611
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:37.934046 11421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:37.934278 11421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:37.934301 11421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:37.935498 11421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:54:37.935544 11421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:37.935559 11421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:37.935570 11421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:54:37.935580 11421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:54:37.937062 11421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:43611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:37.937310 11421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:37.937536 11421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:37.942289 11429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:37.942297 11427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:37.942324 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:37.942494 11421 server_base.cc:1034] running on GCE node
I20260927 03:54:37.942752 11421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:37.943025 11421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:37.944167 11421 hybrid_clock.cc:648] HybridClock initialized: now 1790481277944151 us; error 29 us; skew 500 ppm
Sep 27 03:54:37 dist-test-slave-96fg krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481277, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:37.946521 11421 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:54:37.946900 11421 webserver.cc:460] Webserver started at http://127.1.50.126:44719/ using document root <none> and password file <none>
I20260927 03:54:37.947086 11421 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:37.947124 11421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:37.947212 11421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:37.948230 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "9c1a50214c154f5d988d6bc7bd895cd4"
format_stamp: "Formatted at 2026-09-27 03:54:37 on dist-test-slave-96fg"
server_key: "896be7301ab943ff844bd8f1cb2b00a6"
server_key_iv: "549e78f519a498764e610d997bfc99eb"
server_key_version: "encryptionkey@0"
I20260927 03:54:37.948514 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "9c1a50214c154f5d988d6bc7bd895cd4"
format_stamp: "Formatted at 2026-09-27 03:54:37 on dist-test-slave-96fg"
server_key: "896be7301ab943ff844bd8f1cb2b00a6"
server_key_iv: "549e78f519a498764e610d997bfc99eb"
server_key_version: "encryptionkey@0"
I20260927 03:54:37.949632 11421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:37.950273 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:37.950398 11421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:37.950472 11421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9c1a50214c154f5d988d6bc7bd895cd4"
format_stamp: "Formatted at 2026-09-27 03:54:37 on dist-test-slave-96fg"
server_key: "896be7301ab943ff844bd8f1cb2b00a6"
server_key_iv: "549e78f519a498764e610d997bfc99eb"
server_key_version: "encryptionkey@0"
I20260927 03:54:37.950534 11421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:37.959795 11421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:37.961881 11421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:37.961998 11421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:37.965651 11421 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:43611
I20260927 03:54:37.965687 11488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:43611 every 8 connection(s)
I20260927 03:54:37.966097 11421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:54:37.967146 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:37.968392 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11421
I20260927 03:54:37.968488 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:54:37.968663 1225 external_mini_cluster.cc:1428] Setting key a341cd1a309369d5ae61f2dbe1012a8c
I20260927 03:54:37.969745 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: Bootstrap starting.
I20260927 03:54:37.970345 11489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:37.970597 11489 log.cc:824] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:37.971187 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: No bootstrap required, opened a new log
I20260927 03:54:37.972886 11489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } }
I20260927 03:54:37.973018 11489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:37.973049 11489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1a50214c154f5d988d6bc7bd895cd4, State: Initialized, Role: FOLLOWER
I20260927 03:54:37.973171 11489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } }
I20260927 03:54:37.973254 11489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:54:37.973285 11489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:54:37.973331 11489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:37.973836 11489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } }
I20260927 03:54:37.973927 11489 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c1a50214c154f5d988d6bc7bd895cd4; no voters:
I20260927 03:54:37.974113 11489 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:54:37.974144 11494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:37.974263 11494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1a50214c154f5d988d6bc7bd895cd4, State: Running, Role: LEADER
I20260927 03:54:37.974370 11489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:54:37.974427 11494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } }
Sep 27 03:54:37 dist-test-slave-96fg krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:37.974748 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1a50214c154f5d988d6bc7bd895cd4. Latest consensus state: current_term: 1 leader_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } } }
I20260927 03:54:37.974855 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:37.975063 11501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:54:37.975200 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1a50214c154f5d988d6bc7bd895cd4" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43611 } } }
I20260927 03:54:37.975335 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:54:37.975921 11501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:54:37.977567 11501 catalog_manager.cc:1347] Generated new cluster ID: ec10c7c0491f4bf09ac699f3e8fcd0de
I20260927 03:54:37.977633 11501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:54:37.978884 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:37.969182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.1.50.126:43611)
0927 03:54:37.969385 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:37.969393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:37.969417 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:54:37.969803 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:37.969812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:37.969875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:37.970003 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:37.970638 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:37.971604 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:37.972314 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:37.972700 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:37.972792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:37.975274 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:37.975279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:37.975279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:37.975291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:37.977179 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:37.977554 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:37.977557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:37.977558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:37.977715 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:37.977995 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:37.977995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:37.977996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:37.978300 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:37.978435 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:37.978552 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:54:37.978679 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260927 03:54:37.989624 11501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:54:37.990187 11501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:54:37.992672 11501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: Generated new TSK 0
I20260927 03:54:37.992846 11501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:54:38.020598 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:38.099594 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:38.099784 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:38.099814 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:38.100919 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:38.100957 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:38.100984 11515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:54:38.102412 11515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:38.102625 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:38.102833 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:38.107347 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.107414 11515 server_base.cc:1034] running on GCE node
W20260927 03:54:38.107378 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:38.107374 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.107744 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:38.107975 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:38.109122 11515 hybrid_clock.cc:648] HybridClock initialized: now 1790481278109106 us; error 31 us; skew 500 ppm
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:38.111502 11515 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:54:38.111913 11515 webserver.cc:460] Webserver started at http://127.1.50.65:38339/ using document root <none> and password file <none>
I20260927 03:54:38.112106 11515 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:38.112149 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:38.112272 11515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:38.113281 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "9b614ae825f04c11a3d6d98267091ccf"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "fad598157f6473d1d7205093c67a2005"
server_key_iv: "50699e9691370cac066fc0b581153bf6"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.113582 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "9b614ae825f04c11a3d6d98267091ccf"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "fad598157f6473d1d7205093c67a2005"
server_key_iv: "50699e9691370cac066fc0b581153bf6"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.114852 11515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:54:38.115566 11530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.115698 11515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:54:38.115775 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "9b614ae825f04c11a3d6d98267091ccf"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "fad598157f6473d1d7205093c67a2005"
server_key_iv: "50699e9691370cac066fc0b581153bf6"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.115849 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:38.124018 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:38.126088 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:38.126212 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:38.126442 11515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:38.126730 11515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:38.126767 11515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.126796 11515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:38.126814 11515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.132107 11515 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:42917
I20260927 03:54:38.132177 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:42917 every 8 connection(s)
I20260927 03:54:38.132442 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:54:38.135089 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11515
I20260927 03:54:38.135195 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:54:38.135378 1225 external_mini_cluster.cc:1428] Setting key d0ffb23f554e59fbfd0a7ab9ec500a2f
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:38.142133 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.133166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:32899 (local address 127.1.50.126:43611)
0927 03:54:38.133210 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.133212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.133391 (+ 179us) server_negotiation.cc:407] Connection header received
0927 03:54:38.133933 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.133935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.133944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.133996 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:38.134971 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.135573 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.136912 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.137115 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.137181 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.139495 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:38.139497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:38.139497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:38.139505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:38.140786 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.141161 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.141162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.141162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.141218 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.141448 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.141449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.141449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.141563 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:38.141618 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.141896 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.142000 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:54:38.142278 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.133091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43611 (local address 127.1.50.65:32899)
0927 03:54:38.133272 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.133309 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.133767 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.134093 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.134098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.134639 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:38.134835 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.134840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.135692 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.135693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.136816 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.136819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.137213 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.137215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.137346 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.137848 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:38.137856 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:38.139403 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:38.140860 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.140881 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.140883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.141094 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.141278 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.141279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.141279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.141391 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.141617 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:38.141624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:38.141762 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.141867 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.142050 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260927 03:54:38.142710 11644 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43611
I20260927 03:54:38.142814 11644 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:54:38.143008 11644 heartbeater.cc:511] Master 127.1.50.126:43611 requested a full tablet report, sending...
I20260927 03:54:38.143574 11453 ts_manager.cc:194] Registered new tserver with Master: 9b614ae825f04c11a3d6d98267091ccf (127.1.50.65:42917)
I20260927 03:54:38.144285 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:32899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:54:38.160943 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:38.240279 11649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:38.240481 11649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:38.240504 11649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:38.241616 11649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:38.241676 11649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:38.241703 11649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:54:38.243124 11649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:38.243312 11649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:38.243486 11649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:38.248209 11654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:38.248308 11657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.248344 11649 server_base.cc:1034] running on GCE node
W20260927 03:54:38.248292 11655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.248584 11649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:38.248831 11649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:38.250006 11649 hybrid_clock.cc:648] HybridClock initialized: now 1790481278249996 us; error 31 us; skew 500 ppm
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:38.252432 11649 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:54:38.252919 11649 webserver.cc:460] Webserver started at http://127.1.50.66:36009/ using document root <none> and password file <none>
I20260927 03:54:38.253132 11649 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:38.253199 11649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:38.253331 11649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:38.254458 11649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "dd3ba162b8694fadb4b219495a82d9b4"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "4793b14f4a7d97caaf4d6fcbe53e864b"
server_key_iv: "6d3feeeed713665c9d0494142b80ee34"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.254806 11649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "dd3ba162b8694fadb4b219495a82d9b4"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "4793b14f4a7d97caaf4d6fcbe53e864b"
server_key_iv: "6d3feeeed713665c9d0494142b80ee34"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.256174 11649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:38.257014 11664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.257263 11649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:54:38.257354 11649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "dd3ba162b8694fadb4b219495a82d9b4"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "4793b14f4a7d97caaf4d6fcbe53e864b"
server_key_iv: "6d3feeeed713665c9d0494142b80ee34"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.257437 11649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:38.273595 11649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:38.275799 11649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:38.275931 11649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:38.276141 11649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:38.276430 11649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:38.276463 11649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.276494 11649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:38.276515 11649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.281958 11649 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:40561
I20260927 03:54:38.282007 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:40561 every 8 connection(s)
I20260927 03:54:38.282339 11649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:54:38.285756 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11649
I20260927 03:54:38.285867 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:54:38.286060 1225 external_mini_cluster.cc:1428] Setting key 6db99b656057bde0856745e1cf14ac61
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:38.291910 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.283085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:42715 (local address 127.1.50.126:43611)
0927 03:54:38.283137 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.283138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.283310 (+ 172us) server_negotiation.cc:407] Connection header received
0927 03:54:38.283812 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.283814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.283827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.283870 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:38.284797 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.285313 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.286401 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.286681 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.286745 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.288974 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:38.288975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:38.288976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:38.288983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:38.290455 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.290864 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.290866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.290866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.290930 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.291228 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.291229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.291229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.291377 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:38.291449 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.291720 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.291809 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:54:38.292014 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.283014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43611 (local address 127.1.50.66:42715)
0927 03:54:38.283172 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.283225 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.283720 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.283935 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.283939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.284502 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:38.284691 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.284696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.285392 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.285393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.286322 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.286325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.286765 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.286766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.286899 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.287388 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:38.287396 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:38.288859 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:38.290549 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.290558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.290561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.290790 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.291021 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.291022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.291022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.291149 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.291445 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:38.291450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:38.291574 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.291684 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.291830 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260927 03:54:38.292563 11778 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43611
I20260927 03:54:38.292657 11778 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:38.292836 11778 heartbeater.cc:511] Master 127.1.50.126:43611 requested a full tablet report, sending...
I20260927 03:54:38.293218 11453 ts_manager.cc:194] Registered new tserver with Master: dd3ba162b8694fadb4b219495a82d9b4 (127.1.50.66:40561)
I20260927 03:54:38.293746 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:42715
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:54:38.312237 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:54:38.392542 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:54:38.392745 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:54:38.392765 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:54:38.393868 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:54:38.393903 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:54:38.393927 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:54:38.395341 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:54:38.395535 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:54:38.395713 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:54:38.400715 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:54:38.400719 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.400749 11783 server_base.cc:1034] running on GCE node
W20260927 03:54:38.400731 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:54:38.401125 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:54:38.401364 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:54:38.402531 11783 hybrid_clock.cc:648] HybridClock initialized: now 1790481278402517 us; error 30 us; skew 500 ppm
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:54:38.405133 11783 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:54:38.405612 11783 webserver.cc:460] Webserver started at http://127.1.50.67:41891/ using document root <none> and password file <none>
I20260927 03:54:38.405833 11783 fs_manager.cc:359] Metadata directory not provided
I20260927 03:54:38.405879 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:54:38.406000 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:54:38.407186 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "95b205335da44fe583cd2dee970ef0e2"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "ba88bf591fc17bce4dcd26f35bbde41d"
server_key_iv: "7e4362929296be4647ae6dcc37d0d116"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.407544 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "95b205335da44fe583cd2dee970ef0e2"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "ba88bf591fc17bce4dcd26f35bbde41d"
server_key_iv: "7e4362929296be4647ae6dcc37d0d116"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.409013 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:54:38.409876 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.410066 11783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:54:38.410156 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "95b205335da44fe583cd2dee970ef0e2"
format_stamp: "Formatted at 2026-09-27 03:54:38 on dist-test-slave-96fg"
server_key: "ba88bf591fc17bce4dcd26f35bbde41d"
server_key_iv: "7e4362929296be4647ae6dcc37d0d116"
server_key_version: "encryptionkey@0"
I20260927 03:54:38.410239 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:54:38.420032 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:54:38.422197 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:54:38.422317 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:54:38.422525 11783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:54:38.422793 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:54:38.422824 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.422845 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:54:38.422868 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:54:38.428104 11783 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42557
I20260927 03:54:38.428169 11911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42557 every 8 connection(s)
I20260927 03:54:38.428454 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:54:38.436995 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 11783
I20260927 03:54:38.437104 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:54:38.437306 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.429113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:56499 (local address 127.1.50.126:43611)
0927 03:54:38.429174 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.429176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.429348 (+ 172us) server_negotiation.cc:407] Connection header received
0927 03:54:38.429923 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.429925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.429938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.429989 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:38.430957 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.431479 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.432503 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.432699 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.432754 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.434926 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:38.434928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:38.434928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:38.434936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:38.436008 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.436367 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.436368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.436368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.436420 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.436623 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.436624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.436624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.436732 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:38.436788 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.437149 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.437236 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:54:38.437335 1225 external_mini_cluster.cc:1428] Setting key 90a2957335eb51e467e70cd97197ce37
I20260927 03:54:38.437484 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.429069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43611 (local address 127.1.50.67:56499)
0927 03:54:38.429222 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.429262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.429817 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.430077 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.430082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.430667 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:54:38.430862 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.430867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.431562 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.431563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.432421 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.432423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.432768 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.432769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.432884 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.433385 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:54:38.433393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:54:38.434830 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:54:38.436081 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.436087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.436089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.436301 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.436472 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:54:38.436473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:54:38.436473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:54:38.436570 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:54:38.436791 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:54:38.436797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:54:38.436980 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.437094 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.437262 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260927 03:54:38.438107 11912 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43611
I20260927 03:54:38.438205 11912 heartbeater.cc:464] Registering TS with master...
I20260927 03:54:38.438392 11912 heartbeater.cc:511] Master 127.1.50.126:43611 requested a full tablet report, sending...
I20260927 03:54:38.438884 11453 ts_manager.cc:194] Registered new tserver with Master: 95b205335da44fe583cd2dee970ef0e2 (127.1.50.67:42557)
I20260927 03:54:38.439211 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:56499
I20260927 03:54:38.449489 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:54:38.456391 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.450819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.1.50.126:43611)
0927 03:54:38.450864 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.450866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.450920 (+ 54us) server_negotiation.cc:407] Connection header received
0927 03:54:38.450967 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.450968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.450976 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.451019 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:54:38.451559 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.452047 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.452864 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.453043 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.453095 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.454314 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:54:38.454315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:54:38.454315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:54:38.454322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:54:38.455389 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.455671 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.455672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.455672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.455714 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:54:38.455871 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:54:38.455872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:54:38.455872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:54:38.456032 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:54:38.456121 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.456233 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.456300 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:54:38.458057 11453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:54:38.459612 11453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:54:38.465931 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.461743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:42917 (local address 127.0.0.1:36576)
0927 03:54:38.461873 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.461912 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.461972 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.462433 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.462437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.462439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:38.462565 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.462567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463989 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.463990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.464873 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.464876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.465651 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.465652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.465753 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.465764 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.465822 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.465859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260927 03:54:38.465970 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.462118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.1.50.65:42917)
0927 03:54:38.462286 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.462294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.462303 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:54:38.462335 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.462337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.462352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.462419 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:38.462641 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463875 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.464943 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.465572 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.465644 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.465711 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.465848 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.465889 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260927 03:54:38.466176 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.461754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:40561 (local address 127.0.0.1:47332)
0927 03:54:38.461995 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.462005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.462072 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.462433 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.462437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.462442 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:38.462507 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.462509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463926 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.463927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.465080 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.465082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.465931 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.465932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.465985 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.465988 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.466073 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.466113 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":31,"thread_start_us":70,"threads_started":1}
I20260927 03:54:38.466466 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.461839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.1.50.66:40561)
0927 03:54:38.462251 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.462255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.462266 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:54:38.462309 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.462311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.462326 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.462410 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:38.462594 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463827 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.465171 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.465853 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.465915 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.465965 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.466351 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.466393 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":79,"threads_started":1}
I20260927 03:54:38.466717 11578 tablet_service.cc:1467] Processing CreateTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 (DEFAULT_TABLE table=test-table [id=b8d8b14b07a84b23978f4775f04def23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:38.467042 11578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688fc5c8b7324403b2682ba7655c53c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:38.467053 11712 tablet_service.cc:1467] Processing CreateTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 (DEFAULT_TABLE table=test-table [id=b8d8b14b07a84b23978f4775f04def23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:38.467373 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688fc5c8b7324403b2682ba7655c53c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:38.467476 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.461946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42557 (local address 127.0.0.1:43988)
0927 03:54:38.462573 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.462584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.462636 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.462834 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.462835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.462837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:38.462878 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.462880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463782 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.463783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.465080 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.465082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.467238 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.467239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.467307 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.467312 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.467367 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.467409 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":24,"thread_start_us":97,"threads_started":1}
I20260927 03:54:38.467701 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.462057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.1.50.67:42557)
0927 03:54:38.462247 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.462252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.462701 (+ 449us) server_negotiation.cc:407] Connection header received
0927 03:54:38.462750 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.462752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.462767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.462839 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:38.462964 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.463658 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.466387 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.467065 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.467134 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.467202 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.467396 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.467493 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260927 03:54:38.468489 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 (DEFAULT_TABLE table=test-table [id=b8d8b14b07a84b23978f4775f04def23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:54:38.468802 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688fc5c8b7324403b2682ba7655c53c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:54:38.470067 11926 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Bootstrap starting.
I20260927 03:54:38.470832 11926 tablet_bootstrap.cc:654] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:38.470992 11927 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: Bootstrap starting.
I20260927 03:54:38.471289 11926 log.cc:824] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:38.471592 11927 tablet_bootstrap.cc:654] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:38.471892 11926 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: No bootstrap required, opened a new log
I20260927 03:54:38.471936 11926 ts_tablet_manager.cc:1397] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:38.472121 11927 log.cc:824] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:38.472450 11930 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Bootstrap starting.
I20260927 03:54:38.472833 11927 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: No bootstrap required, opened a new log
I20260927 03:54:38.472903 11927 ts_tablet_manager.cc:1397] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:54:38.473009 11930 tablet_bootstrap.cc:654] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Neither blocks nor log segments found. Creating new log.
I20260927 03:54:38.473472 11930 log.cc:824] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:54:38.473963 11926 raft_consensus.cc:348] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.474151 11926 raft_consensus.cc:374] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:38.474192 11926 raft_consensus.cc:729] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3ba162b8694fadb4b219495a82d9b4, State: Initialized, Role: FOLLOWER
I20260927 03:54:38.474316 11926 consensus_queue.cc:260] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.474578 11926 ts_tablet_manager.cc:1428] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:38.474602 11778 heartbeater.cc:503] Master 127.1.50.126:43611 was elected leader, sending a full tablet report...
I20260927 03:54:38.474962 11927 raft_consensus.cc:348] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.475132 11927 raft_consensus.cc:374] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:38.475171 11927 raft_consensus.cc:729] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b614ae825f04c11a3d6d98267091ccf, State: Initialized, Role: FOLLOWER
I20260927 03:54:38.475279 11927 consensus_queue.cc:260] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.475512 11644 heartbeater.cc:503] Master 127.1.50.126:43611 was elected leader, sending a full tablet report...
I20260927 03:54:38.475646 11930 tablet_bootstrap.cc:492] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: No bootstrap required, opened a new log
I20260927 03:54:38.475713 11930 ts_tablet_manager.cc:1397] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:54:38.475836 11927 ts_tablet_manager.cc:1428] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:54:38.477432 11930 raft_consensus.cc:348] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.477550 11930 raft_consensus.cc:374] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:54:38.477569 11930 raft_consensus.cc:729] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b205335da44fe583cd2dee970ef0e2, State: Initialized, Role: FOLLOWER
I20260927 03:54:38.477648 11930 consensus_queue.cc:260] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.477875 11930 ts_tablet_manager.cc:1428] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:54:38.477916 11912 heartbeater.cc:503] Master 127.1.50.126:43611 was elected leader, sending a full tablet report...
W20260927 03:54:38.532899 11779 tablet.cc:2369] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:54:38.568878 11933 raft_consensus.cc:482] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:54:38.569005 11933 raft_consensus.cc:504] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.569365 11933 leader_election.cc:288] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd3ba162b8694fadb4b219495a82d9b4 (127.1.50.66:40561), 95b205335da44fe583cd2dee970ef0e2 (127.1.50.67:42557)
I20260927 03:54:38.573308 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.569557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:40561 (local address 127.1.50.65:60103)
0927 03:54:38.569616 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.569629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.569693 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.569972 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.569972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.569974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:38.570069 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.570071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.570981 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.570981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.572247 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.572250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.573084 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.573085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.573140 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.573142 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.573187 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.573218 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":94}
I20260927 03:54:38.573311 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.569818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:34027 (local address 127.1.50.67:42557)
0927 03:54:38.569872 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.569874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.569882 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:54:38.569923 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.569924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.569934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.569976 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:38.570138 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.570878 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.572371 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.572947 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.573035 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.573048 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.573192 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.573236 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:54:38.573311 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.569629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:60103 (local address 127.1.50.66:40561)
0927 03:54:38.569688 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.569690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.569708 (+ 18us) server_negotiation.cc:407] Connection header received
0927 03:54:38.569883 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.569884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.569897 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.569953 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:54:38.570135 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.570915 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.572354 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.573019 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.573087 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.573098 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.573204 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.573242 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:54:38.573323 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.569663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42557 (local address 127.1.50.65:34027)
0927 03:54:38.569832 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.569840 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:54:38.569876 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:54:38.569985 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:54:38.569986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:54:38.569987 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:54:38.570069 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.570071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.570970 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.570971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.572247 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:54:38.572250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.573031 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.573031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:54:38.573111 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.573116 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:54:38.573174 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:54:38.573215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":16,"thread_start_us":113,"threads_started":1}
I20260927 03:54:38.573693 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688fc5c8b7324403b2682ba7655c53c2" candidate_uuid: "9b614ae825f04c11a3d6d98267091ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b205335da44fe583cd2dee970ef0e2" is_pre_election: true
I20260927 03:54:38.573856 11866 raft_consensus.cc:2454] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b614ae825f04c11a3d6d98267091ccf in term 0.
I20260927 03:54:38.573825 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688fc5c8b7324403b2682ba7655c53c2" candidate_uuid: "9b614ae825f04c11a3d6d98267091ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3ba162b8694fadb4b219495a82d9b4" is_pre_election: true
I20260927 03:54:38.573994 11732 raft_consensus.cc:2454] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b614ae825f04c11a3d6d98267091ccf in term 0.
I20260927 03:54:38.574108 11532 leader_election.cc:302] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b205335da44fe583cd2dee970ef0e2, 9b614ae825f04c11a3d6d98267091ccf; no voters:
I20260927 03:54:38.574259 11933 raft_consensus.cc:2781] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:54:38.574337 11933 raft_consensus.cc:482] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:54:38.574357 11933 raft_consensus.cc:3037] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:38.575186 11933 raft_consensus.cc:504] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.575349 11933 leader_election.cc:288] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [CANDIDATE]: Term 1 election: Requested vote from peers dd3ba162b8694fadb4b219495a82d9b4 (127.1.50.66:40561), 95b205335da44fe583cd2dee970ef0e2 (127.1.50.67:42557)
I20260927 03:54:38.575475 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688fc5c8b7324403b2682ba7655c53c2" candidate_uuid: "9b614ae825f04c11a3d6d98267091ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3ba162b8694fadb4b219495a82d9b4"
I20260927 03:54:38.575503 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688fc5c8b7324403b2682ba7655c53c2" candidate_uuid: "9b614ae825f04c11a3d6d98267091ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b205335da44fe583cd2dee970ef0e2"
I20260927 03:54:38.575549 11732 raft_consensus.cc:3037] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:38.575567 11866 raft_consensus.cc:3037] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:54:38.576273 11866 raft_consensus.cc:2454] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b614ae825f04c11a3d6d98267091ccf in term 1.
I20260927 03:54:38.576344 11732 raft_consensus.cc:2454] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b614ae825f04c11a3d6d98267091ccf in term 1.
I20260927 03:54:38.576435 11532 leader_election.cc:302] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b205335da44fe583cd2dee970ef0e2, 9b614ae825f04c11a3d6d98267091ccf; no voters:
I20260927 03:54:38.576527 11933 raft_consensus.cc:2781] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:54:38.576697 11933 raft_consensus.cc:686] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 1 LEADER]: Becoming Leader. State: Replica: 9b614ae825f04c11a3d6d98267091ccf, State: Running, Role: LEADER
I20260927 03:54:38.576792 11933 consensus_queue.cc:237] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } }
I20260927 03:54:38.577665 11452 catalog_manager.cc:5495] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b614ae825f04c11a3d6d98267091ccf (127.1.50.65). New cstate: current_term: 1 leader_uuid: "9b614ae825f04c11a3d6d98267091ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b614ae825f04c11a3d6d98267091ccf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 42917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 } health_report { overall_health: UNKNOWN } } }
I20260927 03:54:38.584111 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.581932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.1.50.65:42917)
0927 03:54:38.581976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:54:38.581978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:54:38.581985 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:54:38.582084 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:54:38.582085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:54:38.582102 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:54:38.582180 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:54:38.582328 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.582769 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.583508 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:54:38.583698 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:54:38.583749 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:54:38.583872 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:54:38.583958 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:54:38.584006 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:54:38.584023 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 03:54:38.584037 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:54:38.585757 11732 raft_consensus.cc:1261] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4 [term 1 FOLLOWER]: Refusing update from remote peer 9b614ae825f04c11a3d6d98267091ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:38.585764 11866 raft_consensus.cc:1261] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 1 FOLLOWER]: Refusing update from remote peer 9b614ae825f04c11a3d6d98267091ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:54:38.586043 11933 consensus_queue.cc:1035] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b205335da44fe583cd2dee970ef0e2" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:38.586175 11936 consensus_queue.cc:1035] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3ba162b8694fadb4b219495a82d9b4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:54:38.588846 11941 mvcc.cc:205] Tried to move back new op lower bound from 7333811317084745728 to 7333811317051170816. Current Snapshot: MvccSnapshot[applied={T|T < 7333811317084745728 or (T in {7333811317084745728})}]
I20260927 03:54:38.590482 11452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:38.590560 11452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:54:38.591276 11452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c1a50214c154f5d988d6bc7bd895cd4: Sending DeleteTablet for 3 replicas of tablet 688fc5c8b7324403b2682ba7655c53c2
I20260927 03:54:38.591591 11578 tablet_service.cc:1514] Processing DeleteTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:38 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:36576
I20260927 03:54:38.591604 11846 tablet_service.cc:1514] Processing DeleteTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:38 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:43988
I20260927 03:54:38.591639 11712 tablet_service.cc:1514] Processing DeleteTablet for tablet 688fc5c8b7324403b2682ba7655c53c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:54:38 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:47332
I20260927 03:54:38.591892 11949 tablet_replica.cc:330] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf: stopping tablet replica
I20260927 03:54:38.591987 11949 raft_consensus.cc:2229] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:54:38.592089 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11515
I20260927 03:54:38.592101 11949 raft_consensus.cc:2258] T 688fc5c8b7324403b2682ba7655c53c2 P 9b614ae825f04c11a3d6d98267091ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:38.592902 11948 tablet_replica.cc:330] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: stopping tablet replica
I20260927 03:54:38.592998 11948 raft_consensus.cc:2229] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:54:38.593081 11948 raft_consensus.cc:2258] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:54:38.593463 11948 ts_tablet_manager.cc:1905] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:54:38.595062 11948 ts_tablet_manager.cc:1918] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:54:38.595132 11948 log.cc:1196] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/688fc5c8b7324403b2682ba7655c53c2
I20260927 03:54:38.595376 11948 ts_tablet_manager.cc:1939] T 688fc5c8b7324403b2682ba7655c53c2 P 95b205335da44fe583cd2dee970ef0e2: Deleting consensus metadata
I20260927 03:54:38.595772 11438 catalog_manager.cc:4841] TS 95b205335da44fe583cd2dee970ef0e2 (127.1.50.67:42557): tablet 688fc5c8b7324403b2682ba7655c53c2 (table test-table [id=b8d8b14b07a84b23978f4775f04def23]) successfully deleted
W20260927 03:54:38.596638 11440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:42917 (error 108)
I20260927 03:54:38.596957 11950 tablet_replica.cc:330] T 688fc5c8b7324403b2682ba7655c53c2 P dd3ba162b8694fadb4b219495a82d9b4: stopping tablet replica
I20260927 03:54:38.596971 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11649
I20260927 03:54:38.597129 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.597000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:42917 (local address 127.0.0.1:36592)
0927 03:54:38.597058 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.597096 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:54:38.598932 11440 catalog_manager.cc:4584] TS 9b614ae825f04c11a3d6d98267091ccf (127.1.50.65:42917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 688fc5c8b7324403b2682ba7655c53c2: Network error: Client connection negotiation failed: client connection to 127.1.50.65:42917: connect: Connection refused (error 111)
I20260927 03:54:38.601583 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11783
I20260927 03:54:38.601656 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:54:38.601546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:40561 (local address 127.0.0.1:47334)
0927 03:54:38.601588 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:54:38.601622 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.66:40561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:54:38.605672 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 11421
2026-09-27T03:54:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1818 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11954](info): set up 2 sockets
Sep 27 03:54:38 dist-test-slave-96fg krb5kdc[11954](info): commencing operation
krb5kdc: starting...
W20260927 03:54:39.554883 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:54:39 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:54:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangeradmin_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangeradmin_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangerlookup_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangerlookup_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
I20260927 03:54:39.720934 1225 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBdoVgk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/postgres -l logfile start
2026-09-27 03:54:42.378 UTC [11984] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:54:42.378 UTC [11984] LOG: listening on IPv4 address "127.1.50.84", port 37969
2026-09-27 03:54:42.382 UTC [11984] LOG: listening on Unix socket "/tmp/.s.PGSQL.37969"
2026-09-27 03:54:42.409 UTC [11987] LOG: database system was shut down at 2026-09-27 03:54:40 UTC
2026-09-27 03:54:42.413 UTC [11984] LOG: database system is ready to accept connections
I20260927 03:54:43.282652 1225 mini_postgres.cc:96] Postgres bound to 37969
2026-09-27 03:54:43.288 UTC [11995] FATAL: database "slave" does not exist
127.1.50.84:37969 - accepting connections
I20260927 03:54:43.289002 1225 mini_ranger.cc:161] Starting Ranger...
I20260927 03:54:43.302923 1225 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:54:43.833432 1225 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:54:43.833508 1225 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin
2026-09-27 03:54:44,098 [I] DB FLAVOR :POSTGRES
2026-09-27 03:54:44,098 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:54:44,098 [I] Checking connection..
2026-09-27 03:54:44,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:54:44,334 [I] Checking connection passed.
2026-09-27 03:54:44,334 [I] --------- Verifying version history table ---------
2026-09-27 03:54:44,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:54:44,579 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:54:44,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:54:44,820 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:54:44,820 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:54:44,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:54:45.070 UTC [12059] WARNING: there is no transaction in progress
2026-09-27 03:54:45,076 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:54:45,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:54:45,318 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:54:45,318 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:54:45,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:54:45,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:54:45,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-96fg.c.gcp-upstream.internal','N') ;"
2026-09-27 03:54:46,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:54:46,330 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:54:46,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:54:46,579 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:54:46,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:54:46,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:54:47,076 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:54:47,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:54:48.112 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.130 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.150 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.167 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.189 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.208 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.273 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.280 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.299 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.311 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.322 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.339 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.349 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.356 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.364 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:48.372 UTC [12167] WARNING: there is no transaction in progress
2026-09-27 03:54:49,108 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:54:49,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:54:49,337 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:54:49,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:54:49,572 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:54:49,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:54:49,810 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:54:49,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:54:50,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:54:50,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-96fg.c.gcp-upstream.internal';"
2026-09-27 03:54:50,568 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:54:50,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:54:50,809 [I] DB_PATCHES have already been applied
I20260927 03:54:50.814905 1225 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBdoVgk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:54:50.814986 1225 mini_ranger.cc:193] Using host: 127.1.50.84
I20260927 03:54:50.819065 1225 mini_ranger.cc:227] Ranger admin URL: http://127.1.50.84:37677
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:54:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:54:51.843778 1225 mini_ranger.cc:160] Time spent starting Ranger: real 8.555s user 0.000s sys 0.009s
Sep 27, 2026 3:55:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:55:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 3:55:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:55:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:55:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:55:13.970543 1225 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangerkms_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/rangerkms_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.50.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.50.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:55:14 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481314, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:55:14.065649 1225 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:55:14.075428 1225 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:55:15.171428 1225 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:55:15.171518 1225 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms
I20260927 03:55:15.171536 1225 mini_ranger_kms.cc:227] Using postgres at 127.1.50.84:37969
2026-09-27 03:55:15,343 [I] DB FLAVOR :POSTGRES
2026-09-27 03:55:15,343 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:55:15,343 [I] Checking connection
2026-09-27 03:55:15,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:55:15,612 [I] connection success
2026-09-27 03:55:15,612 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:55:15,612 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:55:15,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:55:15,854 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:55:15,854 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:55:15,854 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:55:15,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:37969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:55:16,128 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:55:16.625141 1225 mini_ranger_kms.cc:310] Created kms service
I20260927 03:55:16.865969 1225 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:55:16.915596 1225 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:55:16 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481314, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:55:17.362582 1225 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:55:17.362706 1225 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:55:17.362728 1225 mini_ranger_kms.cc:240] Using host: 127.1.50.84
I20260927 03:55:17.366528 1225 mini_ranger_kms.cc:276] Ranger KMS PID: 12347
I20260927 03:55:17.366669 1225 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.50.84:34367
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:55:19.451855 1225 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.386s user 0.007s sys 0.146s
I20260927 03:55:19.451946 1225 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:55:19.452026 1225 mini_ranger_kms.cc:401] 127.1.50.84:34367/kms/v1/keys
Sep 27 03:55:19 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:55:20 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
Sep 27 03:55:20 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
Sep 27 03:55:22 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:55:22.869537 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:45797
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:45797
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:34367/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:55:22.956274 12398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:55:22.956501 12398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:55:22.956523 12398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:55:22.957753 12398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:55:22.957804 12398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:55:22.957818 12398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:55:22.957829 12398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:55:22.957844 12398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:55:22.959846 12398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.50.84:34367/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:45797
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:45797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:55:22.960088 12398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:55:22.960311 12398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:55:22.965533 12406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:22.965680 12403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:22.965533 12404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:55:22.965740 12398 server_base.cc:1034] running on GCE node
I20260927 03:55:22.966194 12398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:55:22.966490 12398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:55:22.967665 12398 hybrid_clock.cc:648] HybridClock initialized: now 1790481322967639 us; error 40 us; skew 500 ppm
Sep 27 03:55:22 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:55:22.970464 12398 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:55:22.970997 12398 webserver.cc:460] Webserver started at http://127.1.50.126:42765/ using document root <none> and password file <none>
I20260927 03:55:22.971211 12398 fs_manager.cc:359] Metadata directory not provided
I20260927 03:55:22.971251 12398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:55:22.971375 12398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:55:22 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:55:23.024037 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "b6282cd15ab4494caf72bab40a484aa2"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "ab76d1d803fdc6db79d7f95825712ba3"
server_key_iv: "136566d6ff46b30ebc71ea2543ef1d01"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.024406 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "b6282cd15ab4494caf72bab40a484aa2"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "ab76d1d803fdc6db79d7f95825712ba3"
server_key_iv: "136566d6ff46b30ebc71ea2543ef1d01"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.025849 12398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:55:23.037803 12413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:23.038039 12398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:55:23.038168 12398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "b6282cd15ab4494caf72bab40a484aa2"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "ab76d1d803fdc6db79d7f95825712ba3"
server_key_iv: "136566d6ff46b30ebc71ea2543ef1d01"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.038313 12398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:55:23.063261 12398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:55:23.064977 12398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:55:23.065099 12398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:55:23.069844 12398 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:45797
I20260927 03:55:23.069890 12465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:45797 every 8 connection(s)
I20260927 03:55:23.070240 12398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:55:23.070890 12466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:55:23.073086 12466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: Bootstrap starting.
I20260927 03:55:23.073606 12466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: Neither blocks nor log segments found. Creating new log.
I20260927 03:55:23.073824 12466 log.cc:824] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:55:23.074503 12466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: No bootstrap required, opened a new log
I20260927 03:55:23.075605 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 12398
I20260927 03:55:23.075699 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:55:23.075853 1225 external_mini_cluster.cc:1428] Setting key 815cfbf229d7ecf153fdd3720f5b0189
I20260927 03:55:23.076681 12466 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } }
I20260927 03:55:23.076838 12466 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:55:23.076872 12466 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6282cd15ab4494caf72bab40a484aa2, State: Initialized, Role: FOLLOWER
I20260927 03:55:23.077032 12466 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } }
I20260927 03:55:23.077121 12466 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:55:23.077162 12466 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:55:23.077230 12466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:55:23.077890 12466 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } }
I20260927 03:55:23.077987 12466 leader_election.cc:302] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6282cd15ab4494caf72bab40a484aa2; no voters:
I20260927 03:55:23.078171 12466 leader_election.cc:288] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:55:23.078210 12471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:55:23.078354 12471 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [term 1 LEADER]: Becoming Leader. State: Replica: b6282cd15ab4494caf72bab40a484aa2, State: Running, Role: LEADER
I20260927 03:55:23.078456 12466 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:55:23.078492 12471 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } }
I20260927 03:55:23.078857 12471 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6282cd15ab4494caf72bab40a484aa2. Latest consensus state: current_term: 1 leader_uuid: "b6282cd15ab4494caf72bab40a484aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } } }
I20260927 03:55:23.078866 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6282cd15ab4494caf72bab40a484aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6282cd15ab4494caf72bab40a484aa2" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45797 } } }
I20260927 03:55:23.080056 12471 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:55:23.080111 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:55:23.079870 12466 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:55:23.080463 12486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:55:23.083256 12486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:55:23 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:55:23.086643 12486 catalog_manager.cc:1347] Generated new cluster ID: 8cb82616cda64d9bacc13ede2bbddd91
I20260927 03:55:23.086694 12486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:55:23.089440 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:23.076363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.1.50.126:45797)
0927 03:55:23.076532 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:23.076539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:23.076565 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:55:23.076678 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:23.076683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:23.076731 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:23.076853 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:55:23.077598 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:23.078912 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:23.079980 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:23.082630 (+ 2650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:23.082859 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:23.085652 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:55:23.085657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:55:23.085657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:55:23.085669 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:55:23.087590 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:23.087953 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:23.087956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:23.087957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:23.088064 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:23.088222 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:23.088222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:23.088223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:23.088524 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:55:23.088648 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:23.089069 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 03:55:23.089205 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 03:55:23.111691 12486 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:55:23.112339 12486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:55:23.116819 12486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: Generated new TSK 0
I20260927 03:55:23.117003 12486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:55:23.814224 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:34367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:55:23.901881 12511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:55:23.902128 12511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:55:23.902168 12511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:55:23.903398 12511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:55:23.903439 12511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:55:23.903473 12511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:55:23.905414 12511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.50.84:34367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:55:23.905661 12511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:55:23.905925 12511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:55:23.910830 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:23.910835 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:23.910835 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:55:23.911610 12511 server_base.cc:1034] running on GCE node
I20260927 03:55:23.911880 12511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:55:23.912169 12511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:55:23.913349 12511 hybrid_clock.cc:648] HybridClock initialized: now 1790481323913322 us; error 38 us; skew 500 ppm
Sep 27 03:55:23 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:55:23.916548 12511 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:55:23.917092 12511 webserver.cc:460] Webserver started at http://127.1.50.65:34403/ using document root <none> and password file <none>
I20260927 03:55:23.917369 12511 fs_manager.cc:359] Metadata directory not provided
I20260927 03:55:23.917431 12511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:55:23.917605 12511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:55:23 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:55:23.935297 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "b41448b7446144ad9284622f72e4749b"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "47b648e24f38574dfef4028353b3ba73"
server_key_iv: "0babdfe6b4787f0aa8d8b8e162c0023d"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.935701 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "b41448b7446144ad9284622f72e4749b"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "47b648e24f38574dfef4028353b3ba73"
server_key_iv: "0babdfe6b4787f0aa8d8b8e162c0023d"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.939757 12511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 03:55:23.956866 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:23.957145 12511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 03:55:23.957305 12511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "b41448b7446144ad9284622f72e4749b"
format_stamp: "Formatted at 2026-09-27 03:55:23 on dist-test-slave-96fg"
server_key: "47b648e24f38574dfef4028353b3ba73"
server_key_iv: "0babdfe6b4787f0aa8d8b8e162c0023d"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:23.957458 12511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:55:24.012295 12511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:55:24.014644 12511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:55:24.014776 12511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:55:24.015004 12511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:55:24.015301 12511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:55:24.015336 12511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.015367 12511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:55:24.015391 12511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.022066 12511 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:40895
I20260927 03:55:24.022607 12511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:55:24.022710 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:40895 every 8 connection(s)
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:55:24.031054 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 12511
I20260927 03:55:24.031144 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:55:24.031334 1225 external_mini_cluster.cc:1428] Setting key 6d9c62c865127d67d4de28a979999059
I20260927 03:55:24.036130 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.023321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45797 (local address 127.1.50.65:55875)
0927 03:55:24.023470 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.023513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.024135 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.024563 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.024568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.024824 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:55:24.025138 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.025144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.026212 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.026213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.027511 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.027514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.028063 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.028065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.028214 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.028836 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:55:24.028847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:55:24.030497 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:55:24.034329 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.034340 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.034343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.034667 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.034920 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.034921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.034921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.035092 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.035386 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:55:24.035393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:55:24.035585 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.035707 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.035884 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260927 03:55:24.036816 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.023373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55875 (local address 127.1.50.126:45797)
0927 03:55:24.024324 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.024327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.024335 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:55:24.024360 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.024361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.024373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.024449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:55:24.025258 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.026113 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.027607 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.027959 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.028024 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.030615 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:55:24.030617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:55:24.030617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:55:24.030625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:55:24.034221 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.034771 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.034772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.034772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.034839 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.035171 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.035172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.035172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.035316 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:55:24.035384 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.036619 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.036715 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":101,"threads_started":1}
I20260927 03:55:24.037586 12642 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45797
I20260927 03:55:24.037704 12642 heartbeater.cc:464] Registering TS with master...
I20260927 03:55:24.037907 12642 heartbeater.cc:511] Master 127.1.50.126:45797 requested a full tablet report, sending...
I20260927 03:55:24.038650 12430 ts_manager.cc:194] Registered new tserver with Master: b41448b7446144ad9284622f72e4749b (127.1.50.65:40895)
I20260927 03:55:24.039557 12430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:55875
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:55:24.061794 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:34367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:55:24.165380 12648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:55:24.165594 12648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:55:24.165630 12648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:55:24.167039 12648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:55:24.167088 12648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:55:24.167125 12648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:55:24.168991 12648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.50.84:34367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:55:24.169282 12648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:55:24.169600 12648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:55:24.174872 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:24.174917 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:24.175179 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:55:24.175706 12648 server_base.cc:1034] running on GCE node
I20260927 03:55:24.176020 12648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:55:24.176275 12648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:55:24.177434 12648 hybrid_clock.cc:648] HybridClock initialized: now 1790481324177424 us; error 28 us; skew 500 ppm
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:55:24.180423 12648 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:55:24.180948 12648 webserver.cc:460] Webserver started at http://127.1.50.66:42619/ using document root <none> and password file <none>
I20260927 03:55:24.181375 12648 fs_manager.cc:359] Metadata directory not provided
I20260927 03:55:24.181464 12648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:55:24.181618 12648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:55:24.210278 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "48def5b01cd64a5cbf93cc943a4a1d9c"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "a67733df1c12484694cb5deff374a9b1"
server_key_iv: "b4b5d250310736c94dea46f6d40aa4d5"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.210741 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "48def5b01cd64a5cbf93cc943a4a1d9c"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "a67733df1c12484694cb5deff374a9b1"
server_key_iv: "b4b5d250310736c94dea46f6d40aa4d5"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.212918 12648 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:55:24.228708 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.228974 12648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:55:24.229060 12648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "48def5b01cd64a5cbf93cc943a4a1d9c"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "a67733df1c12484694cb5deff374a9b1"
server_key_iv: "b4b5d250310736c94dea46f6d40aa4d5"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.229156 12648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:55:24.252223 12648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:55:24.254743 12648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:55:24.254958 12648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:55:24.255259 12648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:55:24.255676 12648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:55:24.255761 12648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.255836 12648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:55:24.255895 12648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.264803 12648 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:44569
I20260927 03:55:24.265345 12648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:55:24.265925 12778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:44569 every 8 connection(s)
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:55:24.275168 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 12648
I20260927 03:55:24.275275 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:55:24.275461 1225 external_mini_cluster.cc:1428] Setting key 8c5d19f53638626cbee177c5d95e839b
I20260927 03:55:24.279846 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.266830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45797 (local address 127.1.50.66:58531)
0927 03:55:24.267408 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.267457 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.268172 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.268586 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.268591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.268870 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:55:24.269177 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.269184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.270281 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.270283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.271644 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.271649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.272214 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.272215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.272371 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.273024 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:55:24.273035 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:55:24.274762 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:55:24.277463 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.277472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.277474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.277879 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.278149 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.278150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.278151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.278342 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.278685 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:55:24.278692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:55:24.278900 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.279015 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.279586 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":85,"threads_started":1}
I20260927 03:55:24.279428 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.267127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:58531 (local address 127.1.50.126:45797)
0927 03:55:24.267185 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.267187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.267555 (+ 368us) server_negotiation.cc:407] Connection header received
0927 03:55:24.268318 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.268320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.268345 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.268378 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:55:24.269320 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.270152 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.271788 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.272095 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.272182 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.274890 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:55:24.274892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:55:24.274892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:55:24.274901 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:55:24.277370 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.277968 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.277969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.277969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.278078 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.278456 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.278457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.278457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.278595 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:55:24.278659 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.279203 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.279327 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:55:24.280987 12779 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45797
I20260927 03:55:24.281088 12779 heartbeater.cc:464] Registering TS with master...
I20260927 03:55:24.281286 12779 heartbeater.cc:511] Master 127.1.50.126:45797 requested a full tablet report, sending...
I20260927 03:55:24.281692 12430 ts_manager.cc:194] Registered new tserver with Master: 48def5b01cd64a5cbf93cc943a4a1d9c (127.1.50.66:44569)
I20260927 03:55:24.282220 12430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:58531
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:55:24.314335 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:34367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:55:24.437048 12785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:55:24.437433 12785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:55:24.437515 12785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:55:24.439422 12785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:55:24.439657 12785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:55:24.439754 12785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:55:24.442173 12785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.50.84:34367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:55:24.442627 12785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:55:24.443034 12785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:55:24.449466 12790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:24.449466 12791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:55:24.449502 12793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:55:24.450614 12785 server_base.cc:1034] running on GCE node
I20260927 03:55:24.450843 12785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:55:24.451115 12785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:55:24.452414 12785 hybrid_clock.cc:648] HybridClock initialized: now 1790481324452396 us; error 37 us; skew 500 ppm
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:55:24.459234 12785 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:55:24.459944 12785 webserver.cc:460] Webserver started at http://127.1.50.67:45553/ using document root <none> and password file <none>
I20260927 03:55:24.460229 12785 fs_manager.cc:359] Metadata directory not provided
I20260927 03:55:24.460305 12785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:55:24.460458 12785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:55:24.475764 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "928b4faa66a94feea92b5672a074855a"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "cbcc5660d88d006a6b42b41dec12c74d"
server_key_iv: "0feea389545c25ae656ba8abd79a67ba"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.476311 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "928b4faa66a94feea92b5672a074855a"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "cbcc5660d88d006a6b42b41dec12c74d"
server_key_iv: "0feea389545c25ae656ba8abd79a67ba"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.478400 12785 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:55:24.500918 12800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.501128 12785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 03:55:24.501220 12785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "928b4faa66a94feea92b5672a074855a"
format_stamp: "Formatted at 2026-09-27 03:55:24 on dist-test-slave-96fg"
server_key: "cbcc5660d88d006a6b42b41dec12c74d"
server_key_iv: "0feea389545c25ae656ba8abd79a67ba"
server_key_version: "kuduclusterkey@0"
I20260927 03:55:24.501330 12785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:55:24.515605 12785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:55:24.517906 12785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:55:24.518072 12785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:55:24.518303 12785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:55:24.518647 12785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:55:24.518699 12785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.518729 12785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:55:24.518750 12785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:55:24.531793 12785 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:36587
I20260927 03:55:24.532384 12785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:55:24.539323 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 12785
I20260927 03:55:24.539443 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:55:24.539623 1225 external_mini_cluster.cc:1428] Setting key e1e67c4af2a72a4041689e37c638ed67
I20260927 03:55:24.542860 12913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:36587 every 8 connection(s)
Sep 27 03:55:24 dist-test-slave-96fg krb5kdc[11954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:55:24.562147 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.543882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:58721 (local address 127.1.50.126:45797)
0927 03:55:24.543937 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.543939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.551464 (+ 7525us) server_negotiation.cc:407] Connection header received
0927 03:55:24.552328 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.552330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.552344 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.552372 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:55:24.553281 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.554088 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.555645 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.555969 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.556013 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.558651 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:55:24.558653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:55:24.558653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:55:24.558661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:55:24.560326 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.560836 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.560837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.560838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.560897 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.561193 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.561194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.561194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.561337 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:55:24.561381 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.561813 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.561919 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:55:24.563045 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.543551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45797 (local address 127.1.50.67:58721)
0927 03:55:24.551295 (+ 7744us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.551374 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.552199 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.552574 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.552578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.552868 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:55:24.553169 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.553175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.554200 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.554201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.555539 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.555541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.556109 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.556110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.556272 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.556824 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:55:24.556834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:55:24.558531 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:55:24.560426 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.560434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.560437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.560746 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.560967 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:55:24.560968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:55:24.560968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:55:24.561112 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:55:24.561446 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:55:24.561452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:55:24.561638 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.562572 (+ 934us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.562792 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7676,"thread_start_us":115,"threads_started":1}
I20260927 03:55:24.563707 12914 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45797
I20260927 03:55:24.563817 12914 heartbeater.cc:464] Registering TS with master...
I20260927 03:55:24.564010 12914 heartbeater.cc:511] Master 127.1.50.126:45797 requested a full tablet report, sending...
I20260927 03:55:24.564533 12430 ts_manager.cc:194] Registered new tserver with Master: 928b4faa66a94feea92b5672a074855a (127.1.50.67:36587)
I20260927 03:55:24.565097 12430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:58721
I20260927 03:55:24.573937 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:55:24.584663 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.575781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.1.50.126:45797)
0927 03:55:24.575829 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.575831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.576096 (+ 265us) server_negotiation.cc:407] Connection header received
0927 03:55:24.576185 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.576186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.576198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.576223 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:55:24.577586 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.578451 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.579739 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.580075 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.580118 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.581740 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:55:24.581741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:55:24.581742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:55:24.581750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:55:24.583428 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.583832 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.583834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.583834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.583890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:55:24.584078 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:55:24.584079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:55:24.584079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:55:24.584210 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:55:24.584279 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.584481 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.584560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:55:24.586717 12430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:55:24.589041 12430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:55:24.609040 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.593444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:44569 (local address 127.0.0.1:42862)
0927 03:55:24.597195 (+ 3751us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.597205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.597250 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.598302 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.598303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.598305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:55:24.598349 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.598350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.602596 (+ 4246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.602597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.603681 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.603684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.608819 (+ 5135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.608819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.608882 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.608886 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.608924 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.608962 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3732,"thread_start_us":63,"threads_started":1}
I20260927 03:55:24.609268 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.592446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:40895 (local address 127.0.0.1:34602)
0927 03:55:24.594105 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.594152 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.594220 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.595288 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.595288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.595290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:55:24.595335 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.595336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.600720 (+ 5384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.600721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.602501 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.602504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.608177 (+ 5673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.608178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.608280 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.608298 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.608379 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.608442 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"thread_start_us":82,"threads_started":1}
I20260927 03:55:24.609951 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.592704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36587 (local address 127.0.0.1:48224)
0927 03:55:24.594683 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.594693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.594740 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.595130 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.595133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.595136 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:55:24.595212 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.595214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.597521 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.597522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.604951 (+ 7429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.604953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.609712 (+ 4759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.609713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.609788 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.609792 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.609838 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.609878 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":84,"threads_started":1}
I20260927 03:55:24.610309 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.593690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.1.50.66:44569)
0927 03:55:24.594872 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.594876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.597307 (+ 2431us) server_negotiation.cc:407] Connection header received
0927 03:55:24.597351 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.597353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.597369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.597436 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:55:24.598431 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.599520 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.605207 (+ 5687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.607996 (+ 2789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.608068 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.608139 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.610183 (+ 2044us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.610230 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":94,"threads_started":1}
I20260927 03:55:24.611037 12713 tablet_service.cc:1467] Processing CreateTablet for tablet 91384f788e724096a3fa04e5d87d1995 (DEFAULT_TABLE table=test-table [id=cea6442c1fbc437eaaa1845aee0c8bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:55:24.611430 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91384f788e724096a3fa04e5d87d1995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:55:24.615984 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.593134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.1.50.65:40895)
0927 03:55:24.594438 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.594442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.594452 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:55:24.594486 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.594488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.594504 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.594580 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:55:24.595410 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.600616 (+ 5206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.605361 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.607218 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.607296 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.607368 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.615834 (+ 8466us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.615906 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":58,"threads_started":1}
I20260927 03:55:24.616658 12576 tablet_service.cc:1467] Processing CreateTablet for tablet 91384f788e724096a3fa04e5d87d1995 (DEFAULT_TABLE table=test-table [id=cea6442c1fbc437eaaa1845aee0c8bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:55:24.617020 12576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91384f788e724096a3fa04e5d87d1995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:55:24.621155 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.592962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.1.50.67:36587)
0927 03:55:24.594366 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.594371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.594899 (+ 528us) server_negotiation.cc:407] Connection header received
0927 03:55:24.594945 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.594947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.594963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.595034 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:55:24.595982 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.597065 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.605078 (+ 8013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.609549 (+ 4471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.609607 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.609679 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.620997 (+ 11318us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.621068 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":76,"threads_started":1}
I20260927 03:55:24.621845 12847 tablet_service.cc:1467] Processing CreateTablet for tablet 91384f788e724096a3fa04e5d87d1995 (DEFAULT_TABLE table=test-table [id=cea6442c1fbc437eaaa1845aee0c8bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:55:24.622227 12847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91384f788e724096a3fa04e5d87d1995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:55:24.626636 12935 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Bootstrap starting.
I20260927 03:55:24.627647 12935 tablet_bootstrap.cc:654] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Neither blocks nor log segments found. Creating new log.
I20260927 03:55:24.628464 12934 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: Bootstrap starting.
I20260927 03:55:24.629196 12934 tablet_bootstrap.cc:654] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: Neither blocks nor log segments found. Creating new log.
I20260927 03:55:24.629490 12934 log.cc:824] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:55:24.629873 12933 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Bootstrap starting.
I20260927 03:55:24.630784 12933 tablet_bootstrap.cc:654] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Neither blocks nor log segments found. Creating new log.
I20260927 03:55:24.631063 12933 log.cc:824] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:55:24.631317 12935 log.cc:824] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:55:24.632079 12934 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: No bootstrap required, opened a new log
I20260927 03:55:24.632148 12934 ts_tablet_manager.cc:1397] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:55:24.634862 12934 raft_consensus.cc:348] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.635053 12934 raft_consensus.cc:374] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:55:24.635089 12934 raft_consensus.cc:729] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41448b7446144ad9284622f72e4749b, State: Initialized, Role: FOLLOWER
I20260927 03:55:24.635231 12934 consensus_queue.cc:260] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.635514 12934 ts_tablet_manager.cc:1428] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:55:24.635632 12642 heartbeater.cc:503] Master 127.1.50.126:45797 was elected leader, sending a full tablet report...
I20260927 03:55:24.637415 12933 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: No bootstrap required, opened a new log
I20260927 03:55:24.637498 12933 ts_tablet_manager.cc:1397] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 03:55:24.639822 12933 raft_consensus.cc:348] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.639989 12933 raft_consensus.cc:374] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:55:24.640018 12933 raft_consensus.cc:729] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48def5b01cd64a5cbf93cc943a4a1d9c, State: Initialized, Role: FOLLOWER
I20260927 03:55:24.640154 12933 consensus_queue.cc:260] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.640412 12933 ts_tablet_manager.cc:1428] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:55:24.640513 12779 heartbeater.cc:503] Master 127.1.50.126:45797 was elected leader, sending a full tablet report...
I20260927 03:55:24.641618 12935 tablet_bootstrap.cc:492] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: No bootstrap required, opened a new log
I20260927 03:55:24.641697 12935 ts_tablet_manager.cc:1397] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260927 03:55:24.643882 12935 raft_consensus.cc:348] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.644052 12935 raft_consensus.cc:374] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:55:24.644080 12935 raft_consensus.cc:729] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928b4faa66a94feea92b5672a074855a, State: Initialized, Role: FOLLOWER
I20260927 03:55:24.644214 12935 consensus_queue.cc:260] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.644610 12935 ts_tablet_manager.cc:1428] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:55:24.644739 12914 heartbeater.cc:503] Master 127.1.50.126:45797 was elected leader, sending a full tablet report...
I20260927 03:55:24.645561 12939 raft_consensus.cc:482] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:55:24.645660 12939 raft_consensus.cc:504] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.647348 12939 leader_election.cc:288] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48def5b01cd64a5cbf93cc943a4a1d9c (127.1.50.66:44569), 928b4faa66a94feea92b5672a074855a (127.1.50.67:36587)
I20260927 03:55:24.654676 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.646921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:57559 (local address 127.1.50.66:44569)
0927 03:55:24.646963 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.646965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.647973 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 03:55:24.648122 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.648122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.648133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.648158 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:55:24.648380 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.649156 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.653676 (+ 4520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.654282 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.654328 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.654341 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.654547 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.654589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:55:24.654938 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.646709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:44569 (local address 127.1.50.65:57559)
0927 03:55:24.647917 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.647929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.648007 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.648222 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.648223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.648225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:55:24.648309 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.648311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.649444 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.649444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.651269 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.651272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.654395 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.654396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.654472 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.654475 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.654808 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.654855 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":94,"threads_started":1}
I20260927 03:55:24.655272 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91384f788e724096a3fa04e5d87d1995" candidate_uuid: "b41448b7446144ad9284622f72e4749b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" is_pre_election: true
I20260927 03:55:24.655457 12717 raft_consensus.cc:2454] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41448b7446144ad9284622f72e4749b in term 0.
I20260927 03:55:24.655730 12527 leader_election.cc:302] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48def5b01cd64a5cbf93cc943a4a1d9c, b41448b7446144ad9284622f72e4749b; no voters:
I20260927 03:55:24.655879 12939 raft_consensus.cc:2781] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:55:24.655933 12939 raft_consensus.cc:482] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:55:24.655961 12939 raft_consensus.cc:3037] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:55:24.656139 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.647239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35121 (local address 127.1.50.67:36587)
0927 03:55:24.647273 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.647275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.650177 (+ 2902us) server_negotiation.cc:407] Connection header received
0927 03:55:24.651379 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.651380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.651391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.651418 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:55:24.651616 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.652338 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.655180 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.655762 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.655806 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.655815 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.656040 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.656076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:55:24.656375 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.647092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36587 (local address 127.1.50.65:35121)
0927 03:55:24.650118 (+ 3026us) negotiation.cc:107] Waiting for socket to connect
0927 03:55:24.650129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:55:24.650203 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:55:24.651479 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:55:24.651480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:55:24.651482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:55:24.651552 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.651553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.652444 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.652445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.653561 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:55:24.653563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.655875 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.655875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:55:24.655950 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.655953 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:55:24.656240 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 03:55:24.656287 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3007,"thread_start_us":62,"threads_started":1}
I20260927 03:55:24.656633 12851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91384f788e724096a3fa04e5d87d1995" candidate_uuid: "b41448b7446144ad9284622f72e4749b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928b4faa66a94feea92b5672a074855a" is_pre_election: true
I20260927 03:55:24.656780 12851 raft_consensus.cc:2454] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41448b7446144ad9284622f72e4749b in term 0.
I20260927 03:55:24.656900 12939 raft_consensus.cc:504] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.657192 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91384f788e724096a3fa04e5d87d1995" candidate_uuid: "b41448b7446144ad9284622f72e4749b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c"
I20260927 03:55:24.657289 12717 raft_consensus.cc:3037] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:55:24.658802 12717 raft_consensus.cc:2454] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41448b7446144ad9284622f72e4749b in term 1.
I20260927 03:55:24.659034 12527 leader_election.cc:302] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48def5b01cd64a5cbf93cc943a4a1d9c, b41448b7446144ad9284622f72e4749b; no voters:
I20260927 03:55:24.659152 12941 raft_consensus.cc:2781] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:55:24.659334 12941 raft_consensus.cc:686] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [term 1 LEADER]: Becoming Leader. State: Replica: b41448b7446144ad9284622f72e4749b, State: Running, Role: LEADER
I20260927 03:55:24.659452 12941 consensus_queue.cc:237] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } }
I20260927 03:55:24.660490 12429 catalog_manager.cc:5495] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b reported cstate change: term changed from 0 to 1, leader changed from <none> to b41448b7446144ad9284622f72e4749b (127.1.50.65). New cstate: current_term: 1 leader_uuid: "b41448b7446144ad9284622f72e4749b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41448b7446144ad9284622f72e4749b" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 } health_report { overall_health: UNKNOWN } } }
I20260927 03:55:24.661849 12851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91384f788e724096a3fa04e5d87d1995" candidate_uuid: "b41448b7446144ad9284622f72e4749b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928b4faa66a94feea92b5672a074855a"
I20260927 03:55:24.661952 12851 raft_consensus.cc:3037] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:55:24.663009 12851 raft_consensus.cc:2454] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41448b7446144ad9284622f72e4749b in term 1.
I20260927 03:55:24.663252 12939 leader_election.cc:288] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [CANDIDATE]: Term 1 election: Requested vote from peers 48def5b01cd64a5cbf93cc943a4a1d9c (127.1.50.66:44569), 928b4faa66a94feea92b5672a074855a (127.1.50.67:36587)
I20260927 03:55:24.678594 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:55:24.674945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.1.50.65:40895)
0927 03:55:24.674987 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:55:24.674988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:55:24.675028 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:55:24.675112 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:55:24.675113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:55:24.675140 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:55:24.675214 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:55:24.675418 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.676212 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.677387 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:55:24.677689 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:55:24.677741 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:55:24.677979 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:55:24.678151 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:55:24.678224 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:55:24.678469 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 03:55:24.678491 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:55:24.680826 12851 raft_consensus.cc:1261] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 1 FOLLOWER]: Refusing update from remote peer b41448b7446144ad9284622f72e4749b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:55:24.681180 12945 consensus_queue.cc:1035] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [LEADER]: Connected to new peer: Peer: permanent_uuid: "928b4faa66a94feea92b5672a074855a" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:55:24.683377 12717 raft_consensus.cc:1261] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 1 FOLLOWER]: Refusing update from remote peer b41448b7446144ad9284622f72e4749b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:55:24.683632 12945 consensus_queue.cc:1035] T 91384f788e724096a3fa04e5d87d1995 P b41448b7446144ad9284622f72e4749b [LEADER]: Connected to new peer: Peer: permanent_uuid: "48def5b01cd64a5cbf93cc943a4a1d9c" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:55:24.688738 12948 mvcc.cc:205] Tried to move back new op lower bound from 7333811505889669120 to 7333811505805918208. Current Snapshot: MvccSnapshot[applied={T|T < 7333811505889669120}]
I20260927 03:55:24.688866 12949 mvcc.cc:205] Tried to move back new op lower bound from 7333811505889669120 to 7333811505805918208. Current Snapshot: MvccSnapshot[applied={T|T < 7333811505889669120}]
I20260927 03:55:24.691619 12951 mvcc.cc:205] Tried to move back new op lower bound from 7333811505889669120 to 7333811505805918208. Current Snapshot: MvccSnapshot[applied={T|T < 7333811505889669120}]
I20260927 03:55:24.694573 12429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:55:24.694679 12429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:55:24.695506 12429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6282cd15ab4494caf72bab40a484aa2: Sending DeleteTablet for 3 replicas of tablet 91384f788e724096a3fa04e5d87d1995
I20260927 03:55:24.695874 12576 tablet_service.cc:1514] Processing DeleteTablet for tablet 91384f788e724096a3fa04e5d87d1995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:55:24 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:34602
I20260927 03:55:24.695963 12847 tablet_service.cc:1514] Processing DeleteTablet for tablet 91384f788e724096a3fa04e5d87d1995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:55:24 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:48224
I20260927 03:55:24.696246 12713 tablet_service.cc:1514] Processing DeleteTablet for tablet 91384f788e724096a3fa04e5d87d1995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:55:24 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42862
I20260927 03:55:24.696875 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 12511
I20260927 03:55:24.698164 12958 tablet_replica.cc:330] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: stopping tablet replica
I20260927 03:55:24.698297 12958 raft_consensus.cc:2229] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:55:24.698418 12958 raft_consensus.cc:2258] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:55:24.698916 12958 ts_tablet_manager.cc:1905] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:55:24.700563 12958 ts_tablet_manager.cc:1918] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:55:24.700639 12958 log.cc:1196] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/91384f788e724096a3fa04e5d87d1995
I20260927 03:55:24.701326 12958 ts_tablet_manager.cc:1939] T 91384f788e724096a3fa04e5d87d1995 P 928b4faa66a94feea92b5672a074855a: Deleting consensus metadata
I20260927 03:55:24.701761 12416 catalog_manager.cc:4841] TS 928b4faa66a94feea92b5672a074855a (127.1.50.67:36587): tablet 91384f788e724096a3fa04e5d87d1995 (table test-table [id=cea6442c1fbc437eaaa1845aee0c8bd3]) successfully deleted
I20260927 03:55:24.702163 12959 tablet_replica.cc:330] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: stopping tablet replica
I20260927 03:55:24.702351 12959 raft_consensus.cc:2229] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:55:24.702507 12959 raft_consensus.cc:2258] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:55:24.703019 12959 ts_tablet_manager.cc:1905] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:55:24.709542 12959 ts_tablet_manager.cc:1918] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:55:24.709723 12959 log.cc:1196] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/91384f788e724096a3fa04e5d87d1995
I20260927 03:55:24.710395 12959 ts_tablet_manager.cc:1939] T 91384f788e724096a3fa04e5d87d1995 P 48def5b01cd64a5cbf93cc943a4a1d9c: Deleting consensus metadata
I20260927 03:55:24.710922 12414 catalog_manager.cc:4841] TS 48def5b01cd64a5cbf93cc943a4a1d9c (127.1.50.66:44569): tablet 91384f788e724096a3fa04e5d87d1995 (table test-table [id=cea6442c1fbc437eaaa1845aee0c8bd3]) successfully deleted
W20260927 03:55:24.711450 12415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:40895 (error 108)
I20260927 03:55:24.712076 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 12648
I20260927 03:55:24.718914 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 12785
I20260927 03:55:24.724288 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 12398
I20260927 03:55:24.730382 1225 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:55:25.065164 1225 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:55:25.065275 1225 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:55:25.400532 1225 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:55:25.400 UTC [11984] LOG: received smart shutdown request
2026-09-27 03:55:25.405 UTC [11984] LOG: background worker "logical replication launcher" (PID 11993) exited with exit code 1
2026-09-27 03:55:25.405 UTC [11988] LOG: shutting down
2026-09-27 03:55:25.612 UTC [11984] LOG: database system is shut down
2026-09-27T03:55:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47095 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:55:25 dist-test-slave-96fg krb5kdc[12969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:55:25 dist-test-slave-96fg krb5kdc[12969](info): set up 2 sockets
Sep 27 03:55:25 dist-test-slave-96fg krb5kdc[12969](info): commencing operation
krb5kdc: starting...
W20260927 03:55:26.666254 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:55:26 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:55:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangeradmin_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangeradmin_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangerlookup_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangerlookup_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
I20260927 03:55:26.837576 1225 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBdoVgk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/postgres -l logfile start
2026-09-27 03:55:29.866 UTC [12999] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:55:29.867 UTC [12999] LOG: listening on IPv4 address "127.1.50.84", port 51709
2026-09-27 03:55:29.871 UTC [12999] LOG: listening on Unix socket "/tmp/.s.PGSQL.51709"
2026-09-27 03:55:29.893 UTC [13002] LOG: database system was shut down at 2026-09-27 03:55:27 UTC
2026-09-27 03:55:29.897 UTC [12999] LOG: database system is ready to accept connections
I20260927 03:55:30.778347 1225 mini_postgres.cc:96] Postgres bound to 51709
2026-09-27 03:55:30.785 UTC [13010] FATAL: database "slave" does not exist
127.1.50.84:51709 - accepting connections
I20260927 03:55:30.786451 1225 mini_ranger.cc:161] Starting Ranger...
I20260927 03:55:30.804473 1225 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:55:31.410596 1225 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:55:31.410691 1225 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin
2026-09-27 03:55:31,694 [I] DB FLAVOR :POSTGRES
2026-09-27 03:55:31,695 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:55:31,695 [I] Checking connection..
2026-09-27 03:55:31,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:55:31,993 [I] Checking connection passed.
2026-09-27 03:55:31,993 [I] --------- Verifying version history table ---------
2026-09-27 03:55:31,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:55:32,272 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:55:32,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:55:32,525 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:55:32,525 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:55:32,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:55:32.784 UTC [13074] WARNING: there is no transaction in progress
2026-09-27 03:55:32,790 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:55:32,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:55:33,062 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:55:33,063 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:55:33,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:55:33,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:55:33,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-96fg.c.gcp-upstream.internal','N') ;"
2026-09-27 03:55:33,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:55:34,065 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:55:34,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:55:34,348 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:55:34,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:55:34,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:55:34,883 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:55:34,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:55:36.065 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.088 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.111 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.136 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.162 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.186 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.268 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.281 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.301 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.315 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.328 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.345 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.358 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.367 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.375 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:36.383 UTC [13182] WARNING: there is no transaction in progress
2026-09-27 03:55:37,238 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:55:37,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:55:37,490 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:55:37,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:55:37,775 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:55:37,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:55:38,051 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:55:38,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:55:38,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:55:38,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-96fg.c.gcp-upstream.internal';"
2026-09-27 03:55:38,823 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:55:38,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:55:39,067 [I] DB_PATCHES have already been applied
I20260927 03:55:39.072952 1225 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBdoVgk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:55:39.073026 1225 mini_ranger.cc:193] Using host: 127.1.50.84
I20260927 03:55:39.076234 1225 mini_ranger.cc:227] Ranger admin URL: http://127.1.50.84:34297
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:55:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:55:40.088011 1225 mini_ranger.cc:160] Time spent starting Ranger: real 9.302s user 0.000s sys 0.009s
Sep 27, 2026 3:56:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:56:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 27, 2026 3:56:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:56:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:56:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:56:02.370581 1225 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.50.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.50.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangerkms_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.50.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/rangerkms_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.50.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.50.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.50.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/HTTP_127.1.50.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:56:02 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:56:02.464879 1225 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:56:02.474764 1225 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:56:03.651082 1225 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:56:03.651180 1225 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms
I20260927 03:56:03.651201 1225 mini_ranger_kms.cc:227] Using postgres at 127.1.50.84:51709
2026-09-27 03:56:03,843 [I] DB FLAVOR :POSTGRES
2026-09-27 03:56:03,843 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:56:03,843 [I] Checking connection
2026-09-27 03:56:03,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:56:04,136 [I] connection success
2026-09-27 03:56:04,136 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:56:04,136 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:56:04,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:56:04,407 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:56:04,408 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:56:04,408 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:56:04,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.50.84:51709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:56:04,729 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:56:05.258787 1225 mini_ranger_kms.cc:310] Created kms service
I20260927 03:56:05.485102 1225 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:56:05.545135 1225 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:56:05 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:06.091562 1225 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:56:06.091681 1225 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBdoVgk/build/release/bin/postgresql.jar:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBdoVgk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:56:06.091698 1225 mini_ranger_kms.cc:240] Using host: 127.1.50.84
I20260927 03:56:06.095358 1225 mini_ranger_kms.cc:276] Ranger KMS PID: 13363
I20260927 03:56:06.095450 1225 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.50.84:35779
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:56:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:56:08.143950 1225 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.679s user 0.003s sys 0.169s
I20260927 03:56:08.144027 1225 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:56:08.144071 1225 mini_ranger_kms.cc:401] 127.1.50.84:35779/kms/v1/keys
Sep 27 03:56:08 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:56:08 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
Sep 27 03:56:09 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.50.84@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
Sep 27 03:56:11 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:56:11.549384 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43841
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:43841
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:11.630362 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:11.630604 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:11.630638 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:11.631922 13413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:11.631978 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:11.631995 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:11.632011 13413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:11.632026 13413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:11.634006 13413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.50.84:35779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:43841
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:11.634339 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:11.634569 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:56:11.640290 13413 server_base.cc:1034] running on GCE node
W20260927 03:56:11.640309 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:11.640324 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:11.640347 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:11.640939 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:11.641271 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:11.642452 13413 hybrid_clock.cc:648] HybridClock initialized: now 1790481371642428 us; error 43 us; skew 500 ppm
Sep 27 03:56:11 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:11.645529 13413 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:11.646240 13413 webserver.cc:460] Webserver started at http://127.1.50.126:41611/ using document root <none> and password file <none>
I20260927 03:56:11.646473 13413 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:11.646519 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:11.646647 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:56:11 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:11.703037 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "e4fb5d23726141d98dc4d8856970d046"
format_stamp: "Formatted at 2026-09-27 03:56:11 on dist-test-slave-96fg"
server_key: "af4d4b2a7f8c5aa1f3277ef9780a2f94"
server_key_iv: "0b6f52ead5b99d554425447a4a614009"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:11.703545 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "e4fb5d23726141d98dc4d8856970d046"
format_stamp: "Formatted at 2026-09-27 03:56:11 on dist-test-slave-96fg"
server_key: "af4d4b2a7f8c5aa1f3277ef9780a2f94"
server_key_iv: "0b6f52ead5b99d554425447a4a614009"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:11.705425 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 03:56:11.722716 13428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:11.722951 13413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:56:11.723047 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "e4fb5d23726141d98dc4d8856970d046"
format_stamp: "Formatted at 2026-09-27 03:56:11 on dist-test-slave-96fg"
server_key: "af4d4b2a7f8c5aa1f3277ef9780a2f94"
server_key_iv: "0b6f52ead5b99d554425447a4a614009"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:11.723152 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:11.785147 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:11.787460 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:11.787622 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:11.792640 13413 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:43841
I20260927 03:56:11.792672 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:43841 every 8 connection(s)
I20260927 03:56:11.793112 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:11.793843 13481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:11.796268 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Bootstrap starting.
I20260927 03:56:11.796446 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 13413
I20260927 03:56:11.796545 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:56:11.796736 1225 external_mini_cluster.cc:1428] Setting key 8567610055a6708bd90d54d3522005be
I20260927 03:56:11.796878 13481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:11.797235 13481 log.cc:824] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:11.797997 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: No bootstrap required, opened a new log
I20260927 03:56:11.799638 13481 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:11.799779 13481 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:11.799811 13481 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fb5d23726141d98dc4d8856970d046, State: Initialized, Role: FOLLOWER
I20260927 03:56:11.799959 13481 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:11.800027 13481 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:11.800057 13481 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:11.800117 13481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:11.800658 13481 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:11.800756 13481 leader_election.cc:302] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4fb5d23726141d98dc4d8856970d046; no voters:
I20260927 03:56:11.800936 13481 leader_election.cc:288] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:56:11.800976 13486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:11.801167 13486 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 LEADER]: Becoming Leader. State: Replica: e4fb5d23726141d98dc4d8856970d046, State: Running, Role: LEADER
I20260927 03:56:11.801276 13481 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:11.801301 13486 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:11.801694 13488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4fb5d23726141d98dc4d8856970d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } } }
I20260927 03:56:11.801765 13488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:11.801985 13487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4fb5d23726141d98dc4d8856970d046. Latest consensus state: current_term: 1 leader_uuid: "e4fb5d23726141d98dc4d8856970d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } } }
I20260927 03:56:11.802112 13487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:11.803149 13490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:11.803519 13481 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:56:11.803670 13490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:56:11 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:11.811985 13490 catalog_manager.cc:1347] Generated new cluster ID: 7cc83b09b4ec46399d84227827a103f8
I20260927 03:56:11.812145 13490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:11.812311 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:11.797315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.1.50.126:43841)
0927 03:56:11.797521 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:11.797531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:11.797563 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:56:11.798188 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:11.798198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:11.798268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:11.798425 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:11.799159 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:11.800945 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:11.802165 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:11.802776 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:11.802883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:11.806196 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:11.806255 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:11.806255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:11.806379 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:11.809550 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:11.810064 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:11.810067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:11.810067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:11.810187 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:11.810412 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:11.810413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:11.810414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:11.810905 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:11.811062 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:11.811614 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 03:56:11.811951 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260927 03:56:11.841034 13490 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:56:11.841943 13490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:11.849613 13490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Generated new TSK 0
I20260927 03:56:11.849974 13490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:56:12.619514 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:12.748646 13526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:12.748951 13526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:12.748991 13526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:12.751061 13526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:12.751147 13526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:12.751201 13526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:12.753664 13526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:12.753952 13526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:12.754365 13526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:12.760594 13532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:12.760617 13531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:12.760639 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:12.761646 13526 server_base.cc:1034] running on GCE node
I20260927 03:56:12.761874 13526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:12.762169 13526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:12.763347 13526 hybrid_clock.cc:648] HybridClock initialized: now 1790481372763328 us; error 34 us; skew 500 ppm
Sep 27 03:56:12 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:12.769837 13526 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:12.770486 13526 webserver.cc:460] Webserver started at http://127.1.50.65:33915/ using document root <none> and password file <none>
I20260927 03:56:12.770740 13526 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:12.770779 13526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:12.770874 13526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:56:12 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:12.785645 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "01759b41204e459b99733184286aa0e6"
format_stamp: "Formatted at 2026-09-27 03:56:12 on dist-test-slave-96fg"
server_key: "c1d55c4f7172222c5b854a0b4db31b0a"
server_key_iv: "4174d45e607a5d10a6098a6e77a2d52b"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:12.786062 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "01759b41204e459b99733184286aa0e6"
format_stamp: "Formatted at 2026-09-27 03:56:12 on dist-test-slave-96fg"
server_key: "c1d55c4f7172222c5b854a0b4db31b0a"
server_key_iv: "4174d45e607a5d10a6098a6e77a2d52b"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:12.787806 13526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:56:12.810158 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:12.810415 13526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:56:12.810509 13526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "01759b41204e459b99733184286aa0e6"
format_stamp: "Formatted at 2026-09-27 03:56:12 on dist-test-slave-96fg"
server_key: "c1d55c4f7172222c5b854a0b4db31b0a"
server_key_iv: "4174d45e607a5d10a6098a6e77a2d52b"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:12.810606 13526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:12.828035 13526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:12.831099 13526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:12.831218 13526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:12.831475 13526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:12.831832 13526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:12.831867 13526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:12.831890 13526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:12.831903 13526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:12.838917 13526 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:38751
I20260927 03:56:12.839291 13526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:56:12.840188 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:38751 every 8 connection(s)
Sep 27 03:56:12 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:12.847810 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 13526
I20260927 03:56:12.847914 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:56:12.848093 1225 external_mini_cluster.cc:1428] Setting key ebff76655b58080671af602167993120
I20260927 03:56:12.848930 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:12.840081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.65:37055)
0927 03:56:12.840328 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:12.840372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:12.840992 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:12.841349 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:12.841353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:12.841566 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:12.841768 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:12.841772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:12.842524 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:12.842525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:12.843322 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:12.843325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:12.843694 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:12.843695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:12.843822 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:12.844336 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:12.844345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:12.845556 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:12.847165 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:12.847172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:12.847175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:12.847471 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:12.847722 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:12.847724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:12.847724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:12.847872 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:12.848179 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:12.848185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:12.848395 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 03:56:12.848496 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:56:12.848663 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
I20260927 03:56:12.849352 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:12.840377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37055 (local address 127.1.50.126:43841)
0927 03:56:12.841143 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:12.841145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:12.841151 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:56:12.841172 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:12.841173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:12.841183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:12.841238 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:12.841864 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:12.842441 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:12.843405 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:12.843607 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:12.843664 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:12.845661 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:12.845663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:12.845663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:12.845671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:12.847078 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:12.847546 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:12.847547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:12.847548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:12.847651 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:12.847947 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:12.847948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:12.847948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:12.848094 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:12.848154 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:12.849164 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 03:56:12.849261 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":68,"threads_started":1}
I20260927 03:56:12.850919 13657 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:12.851027 13657 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:12.851212 13657 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:12.851701 13445 ts_manager.cc:194] Registered new tserver with Master: 01759b41204e459b99733184286aa0e6 (127.1.50.65:38751)
I20260927 03:56:12.852303 13445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:37055
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:56:12.878158 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:56:12 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:56:13.000113 13663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:13.000416 13663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:13.000455 13663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:13.002468 13663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:13.002580 13663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:13.002682 13663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:13.005049 13663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:13.005455 13663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:13.005761 13663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:13.012475 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.012600 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.013121 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.014045 13663 server_base.cc:1034] running on GCE node
I20260927 03:56:13.014317 13663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:13.014614 13663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:13.022046 13663 hybrid_clock.cc:648] HybridClock initialized: now 1790481373021966 us; error 65 us; skew 500 ppm
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:13.026170 13663 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:13.027117 13663 webserver.cc:460] Webserver started at http://127.1.50.66:43759/ using document root <none> and password file <none>
I20260927 03:56:13.027637 13663 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:13.027933 13663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:13.028110 13663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:13.043282 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "fa0b762532cc49f1b9e37c1ee0562424"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "13fe6d6cd53c9114fe158ee4e6c22ae5"
server_key_iv: "3e1514d9e73427e1429b65cdffcc43e4"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.043708 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "fa0b762532cc49f1b9e37c1ee0562424"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "13fe6d6cd53c9114fe158ee4e6c22ae5"
server_key_iv: "3e1514d9e73427e1429b65cdffcc43e4"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.046381 13663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 03:56:13.062227 13680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.062599 13663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:13.062731 13663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "fa0b762532cc49f1b9e37c1ee0562424"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "13fe6d6cd53c9114fe158ee4e6c22ae5"
server_key_iv: "3e1514d9e73427e1429b65cdffcc43e4"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.062840 13663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:13.080071 13663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:13.082508 13663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:13.082680 13663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:13.082948 13663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:13.083305 13663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:13.083349 13663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.083379 13663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:13.083442 13663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.090631 13663 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:43105
I20260927 03:56:13.091078 13663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:56:13.094218 13794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:43105 every 8 connection(s)
I20260927 03:56:13.098191 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 13663
I20260927 03:56:13.098299 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:56:13.098481 1225 external_mini_cluster.cc:1428] Setting key 39d44746ff16bb3ed43fa4cecce800cf
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:13.105051 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.092138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:56637 (local address 127.1.50.126:43841)
0927 03:56:13.092199 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:13.092201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:13.092212 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:13.093069 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:13.093071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:13.093085 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:13.093118 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:13.094103 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.094926 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.096536 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.096897 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.096947 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.099891 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:13.099893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:13.099893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:13.099902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:13.101645 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.103703 (+ 2058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.103705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.103706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.103788 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.104121 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.104122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.104122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.104274 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:13.104327 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:13.104809 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 03:56:13.104926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:56:13.105875 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.091885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.66:56637)
0927 03:56:13.092068 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:13.092122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:13.092917 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:13.093343 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:13.093347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:13.093640 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:13.093948 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.093954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.095097 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.095098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.096413 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.096416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.097067 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.097068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.097234 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.097919 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:13.097932 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:13.099763 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:13.101769 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.101780 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.101783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.102140 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.103865 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.103865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.103866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.104048 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.104362 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:13.104369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:13.104571 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 03:56:13.105411 (+ 840us) client_negotiation.cc:240] Negotiation successful
0927 03:56:13.105611 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260927 03:56:13.106479 13795 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:13.106580 13795 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.107061 13795 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:13.107769 13445 ts_manager.cc:194] Registered new tserver with Master: fa0b762532cc49f1b9e37c1ee0562424 (127.1.50.66:43105)
I20260927 03:56:13.108348 13445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:56637
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:56:13.134212 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:13.252498 13800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:13.252799 13800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:13.252835 13800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:13.254608 13800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:13.254676 13800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:13.254716 13800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:13.256918 13800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:13.257207 13800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:13.257491 13800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:13.263269 13805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.263269 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.263820 13806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.264019 13800 server_base.cc:1034] running on GCE node
I20260927 03:56:13.264248 13800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:13.264552 13800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:13.265728 13800 hybrid_clock.cc:648] HybridClock initialized: now 1790481373265711 us; error 29 us; skew 500 ppm
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:13.268893 13800 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:13.269340 13800 webserver.cc:460] Webserver started at http://127.1.50.67:36413/ using document root <none> and password file <none>
I20260927 03:56:13.269547 13800 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:13.269595 13800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:13.269759 13800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:13.283749 13800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "105e9b4ae53d3d801fbe68ded46400a9"
server_key_iv: "0ba22e81c28a840d003a2077da999d52"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.284291 13800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "105e9b4ae53d3d801fbe68ded46400a9"
server_key_iv: "0ba22e81c28a840d003a2077da999d52"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.285872 13800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:56:13.313006 13815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.315173 13800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 03:56:13.315296 13800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "105e9b4ae53d3d801fbe68ded46400a9"
server_key_iv: "0ba22e81c28a840d003a2077da999d52"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.315403 13800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:13.339944 13800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:13.342625 13800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:13.342865 13800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:13.343209 13800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:13.343664 13800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:13.343747 13800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.343814 13800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:13.343860 13800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.351612 13800 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40959
I20260927 03:56:13.352145 13800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:56:13.352837 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40959 every 8 connection(s)
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:13.361521 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 13800
I20260927 03:56:13.361630 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:56:13.362202 1225 external_mini_cluster.cc:1428] Setting key 3a74b160cf1717aa359442f4fe4e2a83
I20260927 03:56:13.364965 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.354232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:53765 (local address 127.1.50.126:43841)
0927 03:56:13.354277 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:13.354280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:13.354677 (+ 397us) server_negotiation.cc:407] Connection header received
0927 03:56:13.355387 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:13.355388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:13.355403 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:13.355435 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:13.356304 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.357052 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.358527 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.358835 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.358929 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.361566 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:13.361568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:13.361568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:13.361577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:13.363249 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.363794 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.363795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.363796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.363862 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.364165 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.364166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.364166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.364333 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:13.364410 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:13.364780 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 03:56:13.364869 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:56:13.365862 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.353902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.67:53765)
0927 03:56:13.354115 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:13.354577 (+ 462us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:13.355259 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:13.355631 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:13.355635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:13.355897 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:13.356183 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.356190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.357167 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.357168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.358447 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.358450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.358955 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.358956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.359104 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.359731 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:13.359741 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:13.361436 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:13.363343 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.363353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.363356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.363695 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.363932 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.363932 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.363933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.364074 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.364411 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:13.364418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:13.364612 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 03:56:13.365427 (+ 815us) client_negotiation.cc:240] Negotiation successful
0927 03:56:13.365615 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260927 03:56:13.366634 13929 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:13.366750 13929 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.366958 13929 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:13.367385 13445 ts_manager.cc:194] Registered new tserver with Master: cfcd35cc3b8f4af68cc4f760157d4d1d (127.1.50.67:40959)
I20260927 03:56:13.367938 13445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:53765
I20260927 03:56:13.374854 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:56:13.375976 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 13526
I20260927 03:56:13.383175 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 13663
I20260927 03:56:13.388907 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 13800
I20260927 03:56:13.393808 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 13413
I20260927 03:56:13.398329 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43841
--webserver_interface=127.1.50.126
--webserver_port=41611
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:43841
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:13.478777 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:13.478995 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:13.479017 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:13.480182 13932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:13.480225 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:13.480237 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:13.480248 13932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:13.480258 13932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:13.481858 13932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:43841
--ranger_config_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:13.482120 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:13.482352 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:13.487262 13937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.487262 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.487298 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.487313 13932 server_base.cc:1034] running on GCE node
I20260927 03:56:13.487799 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:13.488087 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:13.489257 13932 hybrid_clock.cc:648] HybridClock initialized: now 1790481373489244 us; error 32 us; skew 500 ppm
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:13.491961 13932 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:13.492420 13932 webserver.cc:460] Webserver started at http://127.1.50.126:41611/ using document root <none> and password file <none>
I20260927 03:56:13.492621 13932 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:13.492676 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:13.551884 13932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:13.552675 13948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.552857 13932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:13.552979 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "e4fb5d23726141d98dc4d8856970d046"
format_stamp: "Formatted at 2026-09-27 03:56:11 on dist-test-slave-96fg"
server_key: "af4d4b2a7f8c5aa1f3277ef9780a2f94"
server_key_iv: "0b6f52ead5b99d554425447a4a614009"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.553365 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:13.564802 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:13.566570 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:13.566710 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:13.570791 13932 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:43841
I20260927 03:56:13.570876 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:43841 every 8 connection(s)
I20260927 03:56:13.571209 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:13.572386 14001 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:56:13.572999 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Bootstrap starting.
I20260927 03:56:13.573889 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 13932
I20260927 03:56:13.574220 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:38751
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:43841
--webserver_port=33915
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:13.581593 14001 log.cc:824] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:13.583528 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:56:13.583794 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Bootstrap complete.
I20260927 03:56:13.585981 14001 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:13.586275 14001 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fb5d23726141d98dc4d8856970d046, State: Initialized, Role: FOLLOWER
I20260927 03:56:13.586431 14001 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:13.586493 14001 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:13.586534 14001 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:13.586575 14001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:56:13.587558 14001 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:13.587646 14001 leader_election.cc:302] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4fb5d23726141d98dc4d8856970d046; no voters:
I20260927 03:56:13.587808 14001 leader_election.cc:288] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:56:13.587857 14005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:56:13.587992 14005 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [term 2 LEADER]: Becoming Leader. State: Replica: e4fb5d23726141d98dc4d8856970d046, State: Running, Role: LEADER
I20260927 03:56:13.588085 14001 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:13.588140 14005 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } }
I20260927 03:56:13.588438 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4fb5d23726141d98dc4d8856970d046. Latest consensus state: current_term: 2 leader_uuid: "e4fb5d23726141d98dc4d8856970d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } } }
I20260927 03:56:13.588505 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:13.588716 14006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4fb5d23726141d98dc4d8856970d046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fb5d23726141d98dc4d8856970d046" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 43841 } } }
I20260927 03:56:13.588775 14006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:13.588944 14010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:13.589442 14010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:13.589567 14010 catalog_manager.cc:1259] Loaded cluster ID: 7cc83b09b4ec46399d84227827a103f8
I20260927 03:56:13.589604 14010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:13.589778 14001 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:56:13.592890 14010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:13.593459 14010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Loaded TSK: 0
I20260927 03:56:13.593786 14010 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:56:13.682224 14003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:13.682430 14003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:13.682463 14003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:13.683823 14003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:13.683866 14003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:13.683904 14003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:13.685770 14003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:38751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=33915
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:13.686044 14003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:13.686275 14003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:13.691630 14038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.691781 14041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.691895 14039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.693575 14003 server_base.cc:1034] running on GCE node
I20260927 03:56:13.693939 14003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:13.694252 14003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:13.695807 14003 hybrid_clock.cc:648] HybridClock initialized: now 1790481373695716 us; error 35 us; skew 500 ppm
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:13.699299 14003 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:13.699818 14003 webserver.cc:460] Webserver started at http://127.1.50.65:33915/ using document root <none> and password file <none>
I20260927 03:56:13.700084 14003 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:13.700150 14003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:13.724931 14003 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:13.725579 14049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.725749 14003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:56:13.725842 14003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "01759b41204e459b99733184286aa0e6"
format_stamp: "Formatted at 2026-09-27 03:56:12 on dist-test-slave-96fg"
server_key: "c1d55c4f7172222c5b854a0b4db31b0a"
server_key_iv: "4174d45e607a5d10a6098a6e77a2d52b"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.726435 14003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:13.735030 14003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:13.737325 14003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:13.737560 14003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:13.737866 14003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:13.738243 14003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:13.738283 14003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.738315 14003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:13.738351 14003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.744493 14003 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:38751
I20260927 03:56:13.745080 14003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:56:13.746697 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:38751 every 8 connection(s)
I20260927 03:56:13.749799 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14003
I20260927 03:56:13.750177 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:43105
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:43841
--webserver_port=43759
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:13.758782 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.747328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.65:52895)
0927 03:56:13.747557 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:13.747597 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:13.748106 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:13.748539 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:13.748543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:13.748743 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:13.748933 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.748938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.750291 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.750293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.751548 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.751552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.752516 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.752517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.752646 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.753196 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:13.753206 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:13.754617 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:13.756797 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.756804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.756807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.757098 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.757391 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.757391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.757392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.757522 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.758059 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:13.758065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:13.758229 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 03:56:13.758340 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:56:13.758511 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260927 03:56:13.759555 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.747366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:52895 (local address 127.1.50.126:43841)
0927 03:56:13.747559 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:13.747568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:13.747713 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:56:13.748329 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:13.748333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:13.748378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:13.748491 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:13.749042 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.750163 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.751654 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.752290 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.752395 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.754750 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:13.754755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:13.754756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:13.754768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:13.756693 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.757193 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.757198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.757198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.757316 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.757590 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.757591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.757591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.757928 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:13.758006 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:13.759223 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0927 03:56:13.759356 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260927 03:56:13.760854 14163 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:13.760957 14163 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.761157 14163 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:13.762225 14163 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:43841 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:56:13.762300 14163 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.762368 14163 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:13.762838 14163 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.762908 14163 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:13.763247 14163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:56:13.850291 14167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:13.850690 14167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:13.850790 14167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:13.852900 14167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:13.853053 14167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:13.853142 14167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:13.855790 14167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:43105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=43759
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:13.856331 14167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:13.856671 14167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:13.862051 14175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:13.862120 14173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.862366 14167 server_base.cc:1034] running on GCE node
W20260927 03:56:13.862504 14172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:13.862785 14167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:13.863102 14167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:13.864281 14167 hybrid_clock.cc:648] HybridClock initialized: now 1790481373864254 us; error 43 us; skew 500 ppm
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:13.867527 14167 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:13.868139 14167 webserver.cc:460] Webserver started at http://127.1.50.66:43759/ using document root <none> and password file <none>
I20260927 03:56:13.868417 14167 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:13.868502 14167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:13.886305 14167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 03:56:13.888116 14183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.888309 14167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260927 03:56:13.888389 14167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "fa0b762532cc49f1b9e37c1ee0562424"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "13fe6d6cd53c9114fe158ee4e6c22ae5"
server_key_iv: "3e1514d9e73427e1429b65cdffcc43e4"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:13.888856 14167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:13.901435 14167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:13.903769 14167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:13.903913 14167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:13.904112 14167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:13.904412 14167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:13.904443 14167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.904464 14167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:13.904482 14167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:13.911062 14167 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:43105
I20260927 03:56:13.911510 14167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:56:13.911973 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:43105 every 8 connection(s)
I20260927 03:56:13.918505 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14167
Sep 27 03:56:13 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481373, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:13.918802 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.50.84:35779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:40959
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:43841
--webserver_port=36413
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:13.922443 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.912237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:48413 (local address 127.1.50.126:43841)
0927 03:56:13.912490 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:13.912492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:13.912533 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:56:13.913122 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:13.913124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:13.913141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:13.913190 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:13.913762 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.914575 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.916112 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.916447 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.916489 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.919119 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:13.919121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:13.919121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:13.919129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:13.920750 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.921280 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.921281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.921282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.921343 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:13.921641 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:13.921641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:13.921642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:13.921780 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:13.921823 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:13.922245 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 03:56:13.922341 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260927 03:56:13.922999 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:13.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.66:48413)
0927 03:56:13.912408 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:13.912452 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:13.912989 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:13.913267 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:13.913271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:13.913445 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:13.913655 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.913660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.914702 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.914702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.916006 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:13.916010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:13.916605 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:13.916606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:13.916755 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:13.917380 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:13.917397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:13.919000 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:13.920846 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.920855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.920858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.921177 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.921428 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:13.921428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:13.921429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:13.921568 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:13.921867 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:13.921874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:13.922083 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 03:56:13.922583 (+ 500us) client_negotiation.cc:240] Negotiation successful
0927 03:56:13.922764 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260927 03:56:13.923545 14297 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:13.923640 14297 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.923822 14297 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:13.924435 14297 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:43841 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:56:13.924533 14297 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.924609 14297 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:13.924899 14297 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:13.924990 14297 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:13.925248 14297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:56:14.044345 14300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:14.044721 14300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:14.044809 14300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:14.046679 14300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:14.046795 14300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:14.046880 14300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:14.049203 14300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.50.84:35779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=36413
--tserver_master_addrs=127.1.50.126:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:14.049682 14300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:14.050040 14300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:14.056316 14306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:14.056622 14308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:14.056795 14305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:14.057036 14300 server_base.cc:1034] running on GCE node
I20260927 03:56:14.057308 14300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:14.057557 14300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:14.065016 14300 hybrid_clock.cc:648] HybridClock initialized: now 1790481374064983 us; error 40 us; skew 500 ppm
Sep 27 03:56:14 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481374, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:14.068558 14300 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:14.069106 14300 webserver.cc:460] Webserver started at http://127.1.50.67:36413/ using document root <none> and password file <none>
I20260927 03:56:14.069381 14300 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:14.069450 14300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:56:14 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481374, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:14.094437 14300 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.000s
I20260927 03:56:14.098524 14316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:14.098807 14300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 03:56:14.098909 14300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d"
format_stamp: "Formatted at 2026-09-27 03:56:13 on dist-test-slave-96fg"
server_key: "105e9b4ae53d3d801fbe68ded46400a9"
server_key_iv: "0ba22e81c28a840d003a2077da999d52"
server_key_version: "kuduclusterkey@0"
I20260927 03:56:14.099395 14300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:14.117071 14300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:14.119568 14300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:14.119815 14300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:14.120112 14300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:14.120517 14300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:14.120599 14300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:14.120668 14300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:14.120723 14300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:14.126912 14300 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40959
I20260927 03:56:14.127463 14300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:56:14.128129 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14300
I20260927 03:56:14.128538 14430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40959 every 8 connection(s)
Sep 27 03:56:14 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481374, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:14.137269 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:14.129140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.1.50.126:43841)
0927 03:56:14.129258 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:14.129259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:14.129268 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:14.129295 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:14.129296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:14.129309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:14.129369 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:14.130261 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.131067 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.132186 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.132513 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.132592 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:14.133952 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:14.133953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:14.133954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:14.133961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:14.135991 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:14.136332 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:14.136333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:14.136334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:14.136389 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:14.136586 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:14.136587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:14.136587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:14.136728 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:14.136897 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:14.137075 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:56:14.137159 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260927 03:56:14.144301 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:14.129060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:39005 (local address 127.1.50.126:43841)
0927 03:56:14.129106 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:14.129107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:14.130068 (+ 961us) server_negotiation.cc:407] Connection header received
0927 03:56:14.130757 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:14.130758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:14.130771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:14.130800 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:14.131620 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.132364 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.134055 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.134407 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.134493 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:14.137312 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:14.137313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:14.137314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:14.137322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:14.142502 (+ 5180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:14.143157 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:14.143158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:14.143158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:14.143222 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:14.143555 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:14.143556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:14.143556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:14.143684 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:14.143799 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:14.144147 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:56:14.144225 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:56:14.144531 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:14.128434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:43841 (local address 127.1.50.67:39005)
0927 03:56:14.129894 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:14.129951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:14.130636 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:14.130983 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:14.130988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:14.131213 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:14.131509 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:14.131516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.132490 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.132491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:14.133945 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:14.133949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:14.134548 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:14.134550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:14.134685 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:14.135173 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:14.135184 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:14.136829 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:14.142617 (+ 5788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:14.142632 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:14.142636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:14.143031 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:14.143298 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:14.143299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:14.143299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:14.143467 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:14.143796 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:14.143803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:14.143988 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 03:56:14.144106 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:56:14.144284 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"thread_start_us":77,"threads_started":1}
I20260927 03:56:14.145015 14431 heartbeater.cc:347] Connected to a master server at 127.1.50.126:43841
I20260927 03:56:14.145121 14431 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:14.145329 14431 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:14.146243 14431 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:43841 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:56:14.146332 14431 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:14.146452 14431 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:14.146883 14431 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:14.146965 14431 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
W20260927 03:56:14.147298 14431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 03:56:14 dist-test-slave-96fg krb5kdc[12969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481374, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:14.763439 14163 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:14.763592 14163 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:14.764022 13965 ts_manager.cc:194] Registered new tserver with Master: 01759b41204e459b99733184286aa0e6 (127.1.50.65:38751)
I20260927 03:56:14.765014 13965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:52895
I20260927 03:56:14.925424 14297 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:14.925570 14297 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:14.925861 13965 ts_manager.cc:194] Registered new tserver with Master: fa0b762532cc49f1b9e37c1ee0562424 (127.1.50.66:43105)
I20260927 03:56:14.926488 13965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:48413
Sep 27 03:56:15 dist-test-slave-96fg krb5kdc[12969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481374, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for HTTP/127.1.50.84@KRBTEST.COM
I20260927 03:56:15.147548 14431 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:15.147719 14431 heartbeater.cc:511] Master 127.1.50.126:43841 requested a full tablet report, sending...
I20260927 03:56:15.148244 13965 ts_manager.cc:194] Registered new tserver with Master: cfcd35cc3b8f4af68cc4f760157d4d1d (127.1.50.67:40959)
I20260927 03:56:15.148841 13965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:39005
I20260927 03:56:15.151584 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:56:15.170249 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.154567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.1.50.126:43841)
0927 03:56:15.158151 (+ 3584us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.158154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.162242 (+ 4088us) server_negotiation.cc:407] Connection header received
0927 03:56:15.162339 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.162340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.162362 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.162421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:15.163177 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.163950 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.164981 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.165304 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.165366 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.166926 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:15.166927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:15.166928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:15.166935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:15.168744 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:15.169125 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:15.169127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:15.169127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:15.169185 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:15.169354 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:15.169355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:15.169355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:15.169483 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:15.169569 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.170054 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.170143 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3554,"thread_start_us":61,"threads_started":1}
I20260927 03:56:15.172775 13965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:56:15.175127 13965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:56:15.197155 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.189115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40959 (local address 127.0.0.1:34058)
0927 03:56:15.191282 (+ 2167us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.191291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:15.191325 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:15.192794 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:15.192795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:15.192797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:15.192846 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.192848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.194144 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.194145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.195814 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.195817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.196826 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.196827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.196914 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.196928 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:56:15.196998 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:56:15.197057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2152,"thread_start_us":64,"threads_started":1}
I20260927 03:56:15.197489 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.189488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.1.50.65:38751)
0927 03:56:15.190847 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.190853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.190863 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:56:15.190896 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.190898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.190913 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.190976 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:15.191418 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.194906 (+ 3488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.196280 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.196955 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.197011 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.197061 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.197374 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.197419 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":62,"threads_started":1}
I20260927 03:56:15.197826 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.189633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.1.50.67:40959)
0927 03:56:15.192534 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.192540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.192552 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:56:15.192587 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.192589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.192606 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.192697 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:15.192930 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.194060 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.195906 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.196652 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.196727 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.196791 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.197691 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.197741 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2863,"thread_start_us":70,"threads_started":1}
I20260927 03:56:15.197311 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.188809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:38751 (local address 127.0.0.1:59450)
0927 03:56:15.190585 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.190638 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:15.190716 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:15.191063 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:15.191066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:15.191067 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:15.191189 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.191192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.195180 (+ 3988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.195181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.196192 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.196194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.197092 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.197093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.197148 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.197152 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:15.197208 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:56:15.197247 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"thread_start_us":87,"threads_started":1}
I20260927 03:56:15.198798 14357 tablet_service.cc:1467] Processing CreateTablet for tablet eb044d5290f04dd09afe103862135dc8 (DEFAULT_TABLE table=test-table [id=ed4f0f41b24949869f95ca1c3f02adb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:15.198807 14097 tablet_service.cc:1467] Processing CreateTablet for tablet eb044d5290f04dd09afe103862135dc8 (DEFAULT_TABLE table=test-table [id=ed4f0f41b24949869f95ca1c3f02adb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:15.199177 14097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb044d5290f04dd09afe103862135dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:15.199177 14357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb044d5290f04dd09afe103862135dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:15.202991 14466 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Bootstrap starting.
I20260927 03:56:15.203573 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.189908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:43105 (local address 127.0.0.1:53386)
0927 03:56:15.195053 (+ 5145us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.195063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:15.195102 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:15.199392 (+ 4290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:15.199394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:15.199397 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:15.199443 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.199445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.200728 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.200729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.202276 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.202280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.203327 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.203328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.203394 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.203400 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:56:15.203455 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:56:15.203496 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5124,"thread_start_us":64,"threads_started":1}
I20260927 03:56:15.203613 14466 tablet_bootstrap.cc:654] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:15.203981 14466 log.cc:824] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:15.204061 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.190187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.1.50.66:43105)
0927 03:56:15.198138 (+ 7951us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.198144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.198160 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:56:15.198198 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.198200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.198219 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.199288 (+ 1069us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:15.199523 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.200632 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.202379 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.203222 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.203306 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.203364 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.203934 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.203986 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7906,"thread_start_us":63,"threads_started":1}
I20260927 03:56:15.204365 14465 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Bootstrap starting.
I20260927 03:56:15.204795 14231 tablet_service.cc:1467] Processing CreateTablet for tablet eb044d5290f04dd09afe103862135dc8 (DEFAULT_TABLE table=test-table [id=ed4f0f41b24949869f95ca1c3f02adb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:15.205039 14231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb044d5290f04dd09afe103862135dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:15.205834 14465 tablet_bootstrap.cc:654] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:15.206188 14465 log.cc:824] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:15.207358 14466 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: No bootstrap required, opened a new log
I20260927 03:56:15.207479 14466 ts_tablet_manager.cc:1397] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:56:15.208590 14469 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Bootstrap starting.
I20260927 03:56:15.209163 14469 tablet_bootstrap.cc:654] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:15.209443 14469 log.cc:824] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:15.210785 14466 raft_consensus.cc:348] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.210968 14466 raft_consensus.cc:374] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:15.210999 14466 raft_consensus.cc:729] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01759b41204e459b99733184286aa0e6, State: Initialized, Role: FOLLOWER
I20260927 03:56:15.211113 14466 consensus_queue.cc:260] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.211383 14466 ts_tablet_manager.cc:1428] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 03:56:15.211419 14163 heartbeater.cc:503] Master 127.1.50.126:43841 was elected leader, sending a full tablet report...
I20260927 03:56:15.212085 14465 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: No bootstrap required, opened a new log
I20260927 03:56:15.212155 14465 ts_tablet_manager.cc:1397] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260927 03:56:15.214413 14465 raft_consensus.cc:348] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.214596 14465 raft_consensus.cc:374] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:15.214632 14465 raft_consensus.cc:729] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcd35cc3b8f4af68cc4f760157d4d1d, State: Initialized, Role: FOLLOWER
I20260927 03:56:15.214751 14465 consensus_queue.cc:260] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.215020 14465 ts_tablet_manager.cc:1428] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:56:15.215058 14431 heartbeater.cc:503] Master 127.1.50.126:43841 was elected leader, sending a full tablet report...
I20260927 03:56:15.216125 14469 tablet_bootstrap.cc:492] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: No bootstrap required, opened a new log
I20260927 03:56:15.216208 14469 ts_tablet_manager.cc:1397] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 03:56:15.218420 14469 raft_consensus.cc:348] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.218613 14469 raft_consensus.cc:374] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:15.218642 14469 raft_consensus.cc:729] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0b762532cc49f1b9e37c1ee0562424, State: Initialized, Role: FOLLOWER
I20260927 03:56:15.218761 14469 consensus_queue.cc:260] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.219038 14469 ts_tablet_manager.cc:1428] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:56:15.219084 14297 heartbeater.cc:503] Master 127.1.50.126:43841 was elected leader, sending a full tablet report...
W20260927 03:56:15.248178 14164 tablet.cc:2369] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:15.281666 14471 raft_consensus.cc:482] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:15.281870 14471 raft_consensus.cc:504] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.282276 14471 leader_election.cc:288] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa0b762532cc49f1b9e37c1ee0562424 (127.1.50.66:43105), cfcd35cc3b8f4af68cc4f760157d4d1d (127.1.50.67:40959)
I20260927 03:56:15.286480 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.282671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:41689 (local address 127.1.50.66:43105)
0927 03:56:15.282714 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.282717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.282872 (+ 155us) server_negotiation.cc:407] Connection header received
0927 03:56:15.282964 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.282965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.282977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.283006 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:15.283234 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.283990 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.285507 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.286078 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.286120 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.286132 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.286363 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.286403 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:56:15.286803 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.282672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35583 (local address 127.1.50.67:40959)
0927 03:56:15.282714 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.282717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.282857 (+ 140us) server_negotiation.cc:407] Connection header received
0927 03:56:15.282964 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.282965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.282977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.283006 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:15.283233 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.283994 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.285868 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.286422 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.286461 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.286470 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.286684 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.286731 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:56:15.287112 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.282478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:43105 (local address 127.1.50.65:41689)
0927 03:56:15.282825 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.282833 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:15.282892 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:15.283076 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:15.283076 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:15.283077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:15.283158 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.283161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.284097 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.284098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.285402 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.285406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.286206 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.286207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.286287 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.286290 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:56:15.286830 (+ 540us) client_negotiation.cc:240] Negotiation successful
0927 03:56:15.287013 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":98,"thread_start_us":93,"threads_started":1}
I20260927 03:56:15.287117 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.282478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40959 (local address 127.1.50.65:35583)
0927 03:56:15.282786 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.282798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:15.282886 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:15.283073 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:15.283074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:15.283076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:15.283158 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.283161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.284097 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.284098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.285764 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:15.285766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.286522 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.286523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:15.286600 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.286604 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:15.286911 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 03:56:15.287013 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":89,"thread_start_us":91,"threads_started":1}
I20260927 03:56:15.287582 14242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb044d5290f04dd09afe103862135dc8" candidate_uuid: "01759b41204e459b99733184286aa0e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0b762532cc49f1b9e37c1ee0562424" is_pre_election: true
I20260927 03:56:15.287582 14366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb044d5290f04dd09afe103862135dc8" candidate_uuid: "01759b41204e459b99733184286aa0e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" is_pre_election: true
I20260927 03:56:15.287744 14242 raft_consensus.cc:2454] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01759b41204e459b99733184286aa0e6 in term 0.
I20260927 03:56:15.287744 14366 raft_consensus.cc:2454] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01759b41204e459b99733184286aa0e6 in term 0.
I20260927 03:56:15.287989 14052 leader_election.cc:302] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01759b41204e459b99733184286aa0e6, cfcd35cc3b8f4af68cc4f760157d4d1d; no voters:
I20260927 03:56:15.288141 14471 raft_consensus.cc:2781] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:56:15.288190 14471 raft_consensus.cc:482] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:56:15.288213 14471 raft_consensus.cc:3037] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:15.289425 14471 raft_consensus.cc:504] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.289594 14471 leader_election.cc:288] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [CANDIDATE]: Term 1 election: Requested vote from peers fa0b762532cc49f1b9e37c1ee0562424 (127.1.50.66:43105), cfcd35cc3b8f4af68cc4f760157d4d1d (127.1.50.67:40959)
I20260927 03:56:15.289942 14366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb044d5290f04dd09afe103862135dc8" candidate_uuid: "01759b41204e459b99733184286aa0e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d"
I20260927 03:56:15.290086 14366 raft_consensus.cc:3037] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:15.290336 14242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb044d5290f04dd09afe103862135dc8" candidate_uuid: "01759b41204e459b99733184286aa0e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0b762532cc49f1b9e37c1ee0562424"
I20260927 03:56:15.290419 14242 raft_consensus.cc:3037] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:15.290967 14366 raft_consensus.cc:2454] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01759b41204e459b99733184286aa0e6 in term 1.
I20260927 03:56:15.291147 14052 leader_election.cc:302] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01759b41204e459b99733184286aa0e6, cfcd35cc3b8f4af68cc4f760157d4d1d; no voters:
I20260927 03:56:15.291230 14471 raft_consensus.cc:2781] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:15.291215 14242 raft_consensus.cc:2454] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01759b41204e459b99733184286aa0e6 in term 1.
I20260927 03:56:15.291378 14471 raft_consensus.cc:686] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 1 LEADER]: Becoming Leader. State: Replica: 01759b41204e459b99733184286aa0e6, State: Running, Role: LEADER
I20260927 03:56:15.291524 14471 consensus_queue.cc:237] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } }
I20260927 03:56:15.292390 13965 catalog_manager.cc:5495] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01759b41204e459b99733184286aa0e6 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "01759b41204e459b99733184286aa0e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01759b41204e459b99733184286aa0e6" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 } health_report { overall_health: UNKNOWN } } }
I20260927 03:56:15.312191 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.309796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.1.50.65:38751)
0927 03:56:15.309860 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:15.309862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:15.309876 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:56:15.309922 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:15.309924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:15.309951 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:15.310091 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:56:15.310207 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.310736 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.311493 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:15.311690 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:15.311746 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:15.311894 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:56:15.311998 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:56:15.312068 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:15.312088 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 03:56:15.312110 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:56:15.314502 14242 raft_consensus.cc:1261] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 1 FOLLOWER]: Refusing update from remote peer 01759b41204e459b99733184286aa0e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:15.314502 14366 raft_consensus.cc:1261] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 1 FOLLOWER]: Refusing update from remote peer 01759b41204e459b99733184286aa0e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:15.314986 14476 consensus_queue.cc:1035] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0b762532cc49f1b9e37c1ee0562424" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:15.315167 14471 consensus_queue.cc:1035] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcd35cc3b8f4af68cc4f760157d4d1d" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:15.318550 14479 mvcc.cc:205] Tried to move back new op lower bound from 7333811713285496832 to 7333811713194758144. Current Snapshot: MvccSnapshot[applied={T|T < 7333811713285496832}]
I20260927 03:56:15.319875 14480 mvcc.cc:205] Tried to move back new op lower bound from 7333811713285496832 to 7333811713194758144. Current Snapshot: MvccSnapshot[applied={T|T < 7333811713285496832}]
I20260927 03:56:15.320411 14482 mvcc.cc:205] Tried to move back new op lower bound from 7333811713285496832 to 7333811713194758144. Current Snapshot: MvccSnapshot[applied={T|T < 7333811713285496832}]
I20260927 03:56:15.323014 13965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:15.323120 13965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:15.323846 13965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4fb5d23726141d98dc4d8856970d046: Sending DeleteTablet for 3 replicas of tablet eb044d5290f04dd09afe103862135dc8
I20260927 03:56:15.324183 14097 tablet_service.cc:1514] Processing DeleteTablet for tablet eb044d5290f04dd09afe103862135dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:15 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:59450
I20260927 03:56:15.324220 14357 tablet_service.cc:1514] Processing DeleteTablet for tablet eb044d5290f04dd09afe103862135dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:15 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:34058
I20260927 03:56:15.324548 14488 tablet_replica.cc:330] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6: stopping tablet replica
I20260927 03:56:15.324602 14231 tablet_service.cc:1514] Processing DeleteTablet for tablet eb044d5290f04dd09afe103862135dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:15 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:53386
I20260927 03:56:15.324765 14488 raft_consensus.cc:2229] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:56:15.324939 14488 raft_consensus.cc:2258] T eb044d5290f04dd09afe103862135dc8 P 01759b41204e459b99733184286aa0e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:15.325277 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14003
I20260927 03:56:15.326920 14490 tablet_replica.cc:330] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: stopping tablet replica
I20260927 03:56:15.327039 14490 raft_consensus.cc:2229] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:15.327142 14490 raft_consensus.cc:2258] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:15.327605 14490 ts_tablet_manager.cc:1905] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:15.329437 14489 tablet_replica.cc:330] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: stopping tablet replica
I20260927 03:56:15.329583 14489 raft_consensus.cc:2229] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:15.329684 14489 raft_consensus.cc:2258] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:15.330235 14489 ts_tablet_manager.cc:1905] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:15.331476 14490 ts_tablet_manager.cc:1918] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:15.331557 14490 log.cc:1196] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/eb044d5290f04dd09afe103862135dc8
I20260927 03:56:15.331828 14490 ts_tablet_manager.cc:1939] T eb044d5290f04dd09afe103862135dc8 P fa0b762532cc49f1b9e37c1ee0562424: Deleting consensus metadata
I20260927 03:56:15.332244 13950 catalog_manager.cc:4841] TS fa0b762532cc49f1b9e37c1ee0562424 (127.1.50.66:43105): tablet eb044d5290f04dd09afe103862135dc8 (table test-table [id=ed4f0f41b24949869f95ca1c3f02adb0]) successfully deleted
W20260927 03:56:15.332690 13951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:38751 (error 108)
I20260927 03:56:15.333484 14489 ts_tablet_manager.cc:1918] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:15.333534 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:15.333418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:38751 (local address 127.0.0.1:59470)
0927 03:56:15.333468 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:15.333502 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:56:15.333550 14489 log.cc:1196] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/eb044d5290f04dd09afe103862135dc8
I20260927 03:56:15.333788 14489 ts_tablet_manager.cc:1939] T eb044d5290f04dd09afe103862135dc8 P cfcd35cc3b8f4af68cc4f760157d4d1d: Deleting consensus metadata
W20260927 03:56:15.333766 13951 catalog_manager.cc:4584] TS 01759b41204e459b99733184286aa0e6 (127.1.50.65:38751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb044d5290f04dd09afe103862135dc8: Network error: Client connection negotiation failed: client connection to 127.1.50.65:38751: connect: Connection refused (error 111)
I20260927 03:56:15.334064 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14167
I20260927 03:56:15.338688 13951 catalog_manager.cc:4841] TS cfcd35cc3b8f4af68cc4f760157d4d1d (127.1.50.67:40959): tablet eb044d5290f04dd09afe103862135dc8 (table test-table [id=ed4f0f41b24949869f95ca1c3f02adb0]) successfully deleted
I20260927 03:56:15.340445 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14300
I20260927 03:56:15.345500 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 13932
I20260927 03:56:15.350641 1225 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:56:15.695854 1225 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:56:15.695963 1225 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:56:16.030776 1225 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:56:16.030 UTC [12999] LOG: received smart shutdown request
2026-09-27 03:56:16.034 UTC [12999] LOG: background worker "logical replication launcher" (PID 13008) exited with exit code 1
2026-09-27 03:56:16.034 UTC [13003] LOG: shutting down
2026-09-27 03:56:16.256 UTC [12999] LOG: database system is shut down
2026-09-27T03:56:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50645 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:56:16 dist-test-slave-96fg krb5kdc[14500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:56:16 dist-test-slave-96fg krb5kdc[14500](info): set up 2 sockets
Sep 27 03:56:16 dist-test-slave-96fg krb5kdc[14500](info): commencing operation
krb5kdc: starting...
W20260927 03:56:17.326050 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:56:17.457154 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46781
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:17.548816 14513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:17.549073 14513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:17.549104 14513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:17.550349 14513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:17.550400 14513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:17.550418 14513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:17.550436 14513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:17.550449 14513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:17.552013 14513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:17.552273 14513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:17.552500 14513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:56:17.557955 14513 server_base.cc:1034] running on GCE node
W20260927 03:56:17.557955 14521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:17.557981 14519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:17.557981 14518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:17.558624 14513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:17.558919 14513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:17.560109 14513 hybrid_clock.cc:648] HybridClock initialized: now 1790481377560087 us; error 58 us; skew 500 ppm
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:17.562999 14513 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:17.563503 14513 webserver.cc:460] Webserver started at http://127.1.50.126:37071/ using document root <none> and password file <none>
I20260927 03:56:17.563706 14513 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:17.563750 14513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:17.563854 14513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:17.565001 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "1e77e548c4d649fe94d198e6d68dc10e"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "e51e442998ff3500c7932e6c71c71270"
server_key_iv: "6dd67bf98bcc46330d9427e29380a8fc"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.565337 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "1e77e548c4d649fe94d198e6d68dc10e"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "e51e442998ff3500c7932e6c71c71270"
server_key_iv: "6dd67bf98bcc46330d9427e29380a8fc"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.566905 14513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:56:17.567785 14528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:17.568082 14513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:17.568184 14513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "1e77e548c4d649fe94d198e6d68dc10e"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "e51e442998ff3500c7932e6c71c71270"
server_key_iv: "6dd67bf98bcc46330d9427e29380a8fc"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.568256 14513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:17.583060 14513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:17.585279 14513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:17.585438 14513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:17.590547 14513 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:46781
I20260927 03:56:17.590659 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:46781 every 8 connection(s)
I20260927 03:56:17.591023 14513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:17.592083 14581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:17.593192 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14513
I20260927 03:56:17.593322 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:56:17.593515 1225 external_mini_cluster.cc:1428] Setting key cf346e03b2d51f2aedb904465bed385a
I20260927 03:56:17.594280 14581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.125@KRBTEST.COM" created.
I20260927 03:56:17.605625 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.595377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.1.50.126:46781)
0927 03:56:17.595641 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.595653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.595806 (+ 153us) server_negotiation.cc:407] Connection header received
0927 03:56:17.596461 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.596463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.596482 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.596560 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.597138 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.598005 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.598906 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.599225 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.599333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.601775 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.601787 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.601788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.601798 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.603544 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.604064 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.604068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.604069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.604174 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.604488 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.604489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.604489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.604845 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.604925 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.605228 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.605393 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":201,"thread_start_us":148,"threads_started":1}
I20260927 03:56:17.605631 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.595376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46781 (local address 127.0.0.1:35420)
0927 03:56:17.595652 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.595698 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.596289 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.596570 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.596573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.596881 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.597048 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.597052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.598126 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.598127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.598819 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.598830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.599297 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.599298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.599363 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.599866 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.599875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.601633 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.603656 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.603665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.603667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.603938 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.604256 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.604257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.604257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.604359 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.604935 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.604942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.605048 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.605161 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.605496 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":124,"thread_start_us":163,"threads_started":1}
I20260927 03:56:17.606662 14581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.607096 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.606974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50006)
0927 03:56:17.607023 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.607069 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 03:56:17.607227 14531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
I20260927 03:56:17.607784 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.607664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50016)
0927 03:56:17.607727 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.607758 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 03:56:17.607905 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:45321: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
I20260927 03:56:17.621331 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:45321
--webserver_interface=127.1.50.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:17.657160 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } attempt: 1
I20260927 03:56:17.657706 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.657583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50028)
0927 03:56:17.657644 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.657677 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:56:17.658066 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.657953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50038)
0927 03:56:17.657989 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.658012 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 03:56:17.658180 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:45321: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
W20260927 03:56:17.712352 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:17.712622 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:17.712656 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:17.713927 14588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:17.713975 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:17.713994 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:17.714010 14588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:17.714051 14588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:17.715654 14588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.50.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:17.715931 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:17.716169 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:56:17.721494 14588 server_base.cc:1034] running on GCE node
W20260927 03:56:17.721616 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:17.721531 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:17.721565 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:17.722168 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:17.722534 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:17.723721 14588 hybrid_clock.cc:648] HybridClock initialized: now 1790481377723702 us; error 39 us; skew 500 ppm
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:17.726353 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } attempt: 2
I20260927 03:56:17.726667 14588 init.cc:377] Logged in from keytab as kudu/127.1.50.125@KRBTEST.COM (short username kudu)
I20260927 03:56:17.726835 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.726670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50054)
0927 03:56:17.726765 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.726802 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 03:56:17.727280 14588 webserver.cc:460] Webserver started at http://127.1.50.125:40689/ using document root <none> and password file <none>
I20260927 03:56:17.727335 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.727143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50064)
0927 03:56:17.727251 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.727299 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 03:56:17.727495 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:45321: Network error: Client connection negotiation failed: client connection to 127.1.50.125:45321: connect: Connection refused (error 111)
I20260927 03:56:17.727561 14588 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:17.727614 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:17.727742 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:17.729341 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data/instance:
uuid: "0ba0e5bc0486455eb18435a3fb0b38e0"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "a6231e413cc6e457f6a0edec794e745b"
server_key_iv: "3ad7cc1b67add4ae05f4c67e2760e408"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.729725 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal/instance:
uuid: "0ba0e5bc0486455eb18435a3fb0b38e0"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "a6231e413cc6e457f6a0edec794e745b"
server_key_iv: "3ad7cc1b67add4ae05f4c67e2760e408"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.731469 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 03:56:17.732380 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:17.732645 14588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:17.732816 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "0ba0e5bc0486455eb18435a3fb0b38e0"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "a6231e413cc6e457f6a0edec794e745b"
server_key_iv: "3ad7cc1b67add4ae05f4c67e2760e408"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.732934 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:17.763651 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:17.765918 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:17.766150 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:17.771072 14588 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.125:45321
I20260927 03:56:17.771147 14655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.125:45321 every 8 connection(s)
I20260927 03:56:17.771553 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
I20260927 03:56:17.772630 14656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:17.774586 14656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.777359 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14588
I20260927 03:56:17.777467 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-1/wal/instance
I20260927 03:56:17.777647 1225 external_mini_cluster.cc:1428] Setting key 8c09346b16ecce7ddc8ac7c653645e71
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:17.785297 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.775496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.1.50.126:46781)
0927 03:56:17.775592 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.775595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.775836 (+ 241us) server_negotiation.cc:407] Connection header received
0927 03:56:17.776476 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.776478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.776491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.776548 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.777295 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.777946 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.779246 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.779503 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.779611 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.782462 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.782464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.782464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.782473 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.783796 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.784282 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.784283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.784284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.784336 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.784586 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.784587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.784587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.784720 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.784779 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.785089 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.785189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:56:17.785522 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.775446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46781 (local address 127.0.0.1:35422)
0927 03:56:17.775672 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.775718 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.776356 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.776636 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.776640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.776955 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.777199 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.777203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.778078 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.778080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.779121 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.779125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.779622 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.779624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.779802 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.780462 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.780473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.782334 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.783890 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.783905 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.783909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.784209 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.784397 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.784397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.784398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.784525 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.784778 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.784786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.784953 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.785056 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.785259 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.1.50.124@KRBTEST.COM; defaulting to no policy
I20260927 03:56:17.786098 14656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.124@KRBTEST.COM" created.
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:56:17.795713 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.786398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50074)
0927 03:56:17.786444 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.786454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.786492 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.787059 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.787060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.787197 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.787269 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.787270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.788279 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.788280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.788921 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.788924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.789688 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.789690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.789835 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.790334 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.790338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.791961 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.794456 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.794457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.794458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.794670 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.794925 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.794926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.794926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.794983 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.795394 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.795395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.795455 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.795529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.795627 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:56:17.795847 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.786612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.1.50.125:45321)
0927 03:56:17.786815 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.786820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.786850 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:56:17.786881 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.786884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.786900 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.787016 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.787397 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.788161 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.789040 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.789547 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.789657 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.792081 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.792097 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.792097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.792109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.794332 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.794768 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.794773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.794773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.794854 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.795047 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.795048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.795048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.795287 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.795389 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.795590 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.795695 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260927 03:56:17.796710 14656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.797204 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.797038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51610)
0927 03:56:17.797107 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.797150 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 03:56:17.797477 14604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
I20260927 03:56:17.797994 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.797903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51616)
0927 03:56:17.797946 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.797970 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 03:56:17.798125 14656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
I20260927 03:56:17.809844 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:42393
--webserver_interface=127.1.50.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:17.818686 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } attempt: 3
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:56:17.826220 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.819357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.1.50.125:45321)
0927 03:56:17.819432 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.819435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.819446 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:17.819547 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.819548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.819563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.819617 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.820018 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.820520 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.821181 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.821383 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.821449 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.823767 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.823769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.823769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.823775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.825207 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.825512 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.825513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.825514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.825560 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.825716 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.825717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.825717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.825824 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.825880 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.826038 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.826116 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:56:17.826223 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.819225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50084)
0927 03:56:17.819339 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.819357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.819446 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.819641 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.819643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.819853 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.819929 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.819931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.820589 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.820590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.821097 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.821098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.821453 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.821454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.821530 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.822118 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.822123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.823628 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.825291 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.825292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.825293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.825452 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.825612 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.825612 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.825613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.825667 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.825878 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.825879 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.825928 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.825988 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.826116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 03:56:17.827039 14581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.827476 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.827370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51626)
0927 03:56:17.827417 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.827445 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:56:17.827908 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.827823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51642)
0927 03:56:17.827867 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.827892 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:56:17.828001 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
I20260927 03:56:17.847443 14656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 1
I20260927 03:56:17.848112 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.847922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51648)
0927 03:56:17.848022 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.848083 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 03:56:17.848551 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.848465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51656)
0927 03:56:17.848508 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.848531 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:56:17.848641 14656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
I20260927 03:56:17.859161 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 1
I20260927 03:56:17.859647 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.859536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51670)
0927 03:56:17.859588 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.859620 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:56:17.860188 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.860061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51672)
0927 03:56:17.860129 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.860164 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 03:56:17.860306 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
W20260927 03:56:17.902009 14663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:17.902292 14663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:17.902323 14663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:17.903774 14663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:17.903831 14663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:17.903851 14663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:17.903869 14663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:17.903885 14663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:17.905624 14663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:46781,127.1.50.125:45321,127.1.50.124:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.50.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:17.905861 14663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:17.906097 14663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:17.911330 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:17.911520 14671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:17.911651 14663 server_base.cc:1034] running on GCE node
W20260927 03:56:17.911798 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:17.912124 14663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:17.912463 14663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:17.913614 14663 hybrid_clock.cc:648] HybridClock initialized: now 1790481377913572 us; error 61 us; skew 500 ppm
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:17.916513 14663 init.cc:377] Logged in from keytab as kudu/127.1.50.124@KRBTEST.COM (short username kudu)
I20260927 03:56:17.916806 14656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 2
I20260927 03:56:17.917140 14663 webserver.cc:460] Webserver started at http://127.1.50.124:35191/ using document root <none> and password file <none>
I20260927 03:56:17.917276 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.917094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51680)
0927 03:56:17.917183 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.917233 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 03:56:17.917439 14663 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:17.917507 14663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:17.917635 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.917551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51682)
0927 03:56:17.917589 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.917616 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:56:17.917670 14663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260927 03:56:17.917743 14656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
I20260927 03:56:17.918967 14663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data/instance:
uuid: "43e476f03a2946488067649caa24f3eb"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "effad052f4026d481843cee7c334c940"
server_key_iv: "3579a0f7af8d45c7b8a7e93f129a351f"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.919283 14663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal/instance:
uuid: "43e476f03a2946488067649caa24f3eb"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "effad052f4026d481843cee7c334c940"
server_key_iv: "3579a0f7af8d45c7b8a7e93f129a351f"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.920816 14663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:56:17.921862 14678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:17.922259 14663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:56:17.922358 14663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "43e476f03a2946488067649caa24f3eb"
format_stamp: "Formatted at 2026-09-27 03:56:17 on dist-test-slave-96fg"
server_key: "effad052f4026d481843cee7c334c940"
server_key_iv: "3579a0f7af8d45c7b8a7e93f129a351f"
server_key_version: "encryptionkey@0"
I20260927 03:56:17.922444 14663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:17.934567 14663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:17.935449 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 2
I20260927 03:56:17.935992 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.935807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51688)
0927 03:56:17.935893 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.935956 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 03:56:17.936441 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.936320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51696)
0927 03:56:17.936370 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.936419 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:56:17.936538 14581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42393: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42393: connect: Connection refused (error 111)
I20260927 03:56:17.937325 14663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:17.937479 14663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:17.942305 14663 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.124:42393
I20260927 03:56:17.942377 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.124:42393 every 8 connection(s)
I20260927 03:56:17.942762 14663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
I20260927 03:56:17.943799 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:17.945694 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14663
I20260927 03:56:17.945832 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481222547165-1225-0/minicluster-data/master-2/wal/instance
I20260927 03:56:17.945660 14731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.946095 1225 external_mini_cluster.cc:1428] Setting key c5d0fa78de2847623269e4cde91ee36a
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:56:17.956017 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.946590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.1.50.126:46781)
0927 03:56:17.946664 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.946667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.946920 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:56:17.947595 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.947597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.947611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.947663 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.948295 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.948791 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.949745 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.949951 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.950103 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.952764 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.952765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.952766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.952774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.954233 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.954750 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.954753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.954753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.954839 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.955173 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.955174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.955174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.955309 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.955367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.955799 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.955905 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:56:17.956168 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.946518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:46781 (local address 127.0.0.1:35426)
0927 03:56:17.946748 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.946798 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.947465 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.947692 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.947696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.948015 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.948207 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.948212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.948868 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.948870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.949664 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.949667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.950130 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.950132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.950423 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.951022 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.951031 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.952672 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.954344 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.954354 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.954356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.954654 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.954943 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.954944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.954944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.955087 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.955379 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.955386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.955559 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.955756 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.955911 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260927 03:56:17.956597 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.946725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.1.50.124:42393)
0927 03:56:17.946890 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.946924 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.946959 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:56:17.947490 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.947508 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.947545 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.947670 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.948407 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.949293 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.950242 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.950602 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.950675 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.953280 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.953289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.953289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.953297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.955008 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.955436 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.955439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.955439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.955493 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.955697 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.955698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.955699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.955900 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.956094 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.956242 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.956342 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260927 03:56:17.956775 14731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:17.963944 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.957061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:45321 (local address 127.0.0.1:50086)
0927 03:56:17.957160 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.957238 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.957293 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.957493 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.957494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.957607 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.957666 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.957668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.958345 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.958347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.958867 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.958869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.959185 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.959186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.959260 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.960043 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.960047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.961502 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.962996 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.962998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.962998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.963171 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.963336 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.963337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.963337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.963396 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.963617 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.963618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.963665 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.963726 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.963836 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 03:56:17.963943 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.957149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.1.50.125:45321)
0927 03:56:17.957215 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.957218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.957299 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:56:17.957353 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.957354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.957367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.957422 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.957752 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.958278 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.958938 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.959125 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.959181 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.961595 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.961597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.961597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.961604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.962916 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.963231 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.963232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.963233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.963284 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.963444 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.963445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.963445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.963563 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.963618 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.963754 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.963834 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:56:17.964314 14731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:56:17.964907 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.957445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.1.50.126:46781)
0927 03:56:17.957493 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.957494 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.957503 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:17.957621 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.957622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.957633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.957677 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.958164 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.958653 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.959309 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.959662 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.959750 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.962203 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.962205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.962205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.962213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.963579 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.963927 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.963928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.963928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.963977 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.964203 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.964204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.964204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.964337 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.964502 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.964717 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.964804 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
Sep 27 03:56:17 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:56:17.971343 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.964583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51714)
0927 03:56:17.964626 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:17.964637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:17.964671 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:17.964872 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:17.964872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:17.964995 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:17.965045 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.965046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.965781 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.965782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.966373 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:17.966375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.966808 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.966809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:17.966875 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.967309 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:17.967313 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:17.968688 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:17.970155 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.970156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.970156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.970334 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.970592 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:17.970593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:17.970594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:17.970659 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:17.970937 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:17.970938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:17.970990 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 03:56:17.971067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:56:17.971191 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:56:17.971424 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.964659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.1.50.124:42393)
0927 03:56:17.964716 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.964719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.964727 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:56:17.964752 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.964753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.964769 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.964826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.965118 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.965726 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.966437 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.966699 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.966772 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.968763 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.968764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.968764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.968771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.970076 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.970418 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.970420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.970420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.970478 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.970729 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.970730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.970731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.970870 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.970942 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.971114 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.971244 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":27}
I20260927 03:56:17.973201 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: Bootstrap starting.
I20260927 03:56:17.973644 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.965928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.1.50.125:45321)
0927 03:56:17.965970 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.965971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.965980 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:17.965998 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.965998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.966008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.966084 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.966549 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.967071 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.967835 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.968018 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.968072 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.970424 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.970426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.970426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.970436 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.972227 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.972639 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.972640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.972640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.972701 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.973018 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.973019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.973020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.973149 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.973346 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.973426 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.973551 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:56:17.974399 14731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:17.974918 14731 log.cc:824] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:17.975983 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: No bootstrap required, opened a new log
I20260927 03:56:17.976233 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:56:17.978174 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:17.978354 14731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:17.978416 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e476f03a2946488067649caa24f3eb, State: Initialized, Role: FOLLOWER
I20260927 03:56:17.978539 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:17.978856 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:17.980612 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:17.981364 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:56:17.984220 14755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:56:17.984351 14755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:56:17.982085 14663 master_runner.cc:186] Error getting master registration for 127.1.50.126:46781: OK, instance_id { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" instance_seqno: 1790481377589083 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:56:17.986984 14663 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:56:17.987347 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.977834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.1.50.126:46781)
0927 03:56:17.977884 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.977886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.978604 (+ 718us) server_negotiation.cc:407] Connection header received
0927 03:56:17.978748 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.978749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.978762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.978794 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.979513 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.980212 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.981230 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.981541 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.981604 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.983366 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.983367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.983368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.983375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.985772 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.986308 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.986309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.986309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.986376 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.986554 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.986555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.986555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.986703 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.986791 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.987168 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.987250 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:56:17.987455 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.978089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.1.50.124:42393)
0927 03:56:17.980351 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.980353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.980362 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:17.980382 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.980382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.980393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.980465 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.981166 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.981852 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.982870 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.983171 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.983232 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.984479 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.984481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.984482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.984488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.986159 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.986557 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.986558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.986559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.986616 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.986816 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.986816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.986817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.986976 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.987061 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.987277 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.987377 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2240}
I20260927 03:56:17.989053 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:17.978111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.1.50.125:45321)
0927 03:56:17.978154 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:17.978155 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:17.978164 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:17.978269 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:17.978271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:17.978283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:17.978358 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:17.979226 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.979962 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.980987 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:17.981269 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:17.981328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:17.983068 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:17.983069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:17.983070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:17.983077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:17.984968 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.987903 (+ 2935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.987904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.987905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.987969 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:17.988221 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:17.988222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:17.988222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:17.988357 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:17.988851 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:17.988883 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 03:56:17.988987 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:56:18.008910 14656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 3
Sep 27 03:56:18 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:56:18.016381 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:18.009585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.1.50.124:42393)
0927 03:56:18.009661 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:18.009663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:18.009670 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:56:18.009691 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:18.009692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:18.009700 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:18.009745 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:18.010062 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.010547 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.011192 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.011372 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.011429 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:18.013397 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:18.013400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:18.013401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:18.013413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:18.015217 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:18.015672 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:18.015672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:18.015673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:18.015729 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:18.015912 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:18.015913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:18.015913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:18.016021 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:18.016076 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:18.016246 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:56:18.016311 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:56:18.016484 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:18.009444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51724)
0927 03:56:18.009530 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:18.009540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:18.009580 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:18.009762 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:18.009763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:18.009927 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:18.009972 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:18.009974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.010615 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.010616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:18.011116 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:18.011118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.011429 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.011430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:18.011478 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:18.012064 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:18.012068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:18.013281 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:18.015303 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:18.015305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:18.015306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:18.015585 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:18.015787 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:18.015788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:18.015788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:18.015853 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:18.016082 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:18.016083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:18.016149 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 03:56:18.016222 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:56:18.016349 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 03:56:18.018625 14656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: Bootstrap starting.
I20260927 03:56:18.019563 14656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:18.019914 14656 log.cc:824] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:18.021036 14656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: No bootstrap required, opened a new log
I20260927 03:56:18.023763 14656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.023936 14656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:18.023969 14656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba0e5bc0486455eb18435a3fb0b38e0, State: Initialized, Role: FOLLOWER
I20260927 03:56:18.024106 14656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.024497 14656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:18.024477 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.024638 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:56:18.026818 14769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:56:18.026960 14769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:56:18.026976 14588 master_runner.cc:186] Error getting master registration for 127.1.50.126:46781: OK, instance_id { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" instance_seqno: 1790481377589083 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:56:18.028870 14588 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:56:18.035708 14581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } attempt: 3
Sep 27 03:56:18 dist-test-slave-96fg krb5kdc[14500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481377, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:56:18.042510 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:18.036041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42393 (local address 127.0.0.1:51732)
0927 03:56:18.036100 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:18.036114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:18.036208 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:18.036330 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:18.036331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:18.036493 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:18.036541 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:18.036543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.037197 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.037198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:18.037681 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:18.037683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.038065 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.038066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:18.038121 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:18.038564 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:18.038569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:18.040197 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:18.041591 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:18.041592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:18.041592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:18.041747 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:18.041917 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:18.041918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:18.041918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:18.041973 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:18.042204 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:18.042205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:18.042251 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:56:18.042311 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:56:18.042413 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:56:18.042510 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:18.036083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.1.50.124:42393)
0927 03:56:18.036124 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:18.036126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:18.036231 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:56:18.036260 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:18.036261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:18.036271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:18.036329 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:18.036606 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.037125 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.037781 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:18.037969 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:18.038041 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:18.040267 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:18.040269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:18.040269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:18.040277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:18.041516 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:18.041807 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:18.041808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:18.041809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:18.041863 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:18.042045 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:18.042046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:18.042046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:18.042150 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:18.042205 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:18.042339 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:56:18.042413 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:56:18.043843 14581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: Bootstrap starting.
I20260927 03:56:18.044497 14581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:18.044764 14581 log.cc:824] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:18.045470 14581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: No bootstrap required, opened a new log
I20260927 03:56:18.046974 14581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.047094 14581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:18.047127 14581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e77e548c4d649fe94d198e6d68dc10e, State: Initialized, Role: FOLLOWER
I20260927 03:56:18.047258 14581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.047530 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.047603 14581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:18.047667 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:56:18.049351 14783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:56:18.049454 14783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:56:18.052502 14513 master_runner.cc:263] No leader master found from master 1e77e548c4d649fe94d198e6d68dc10e
I20260927 03:56:18.052570 14513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:56:18.086506 14534 tablet.cc:2369] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:18.171711 14744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:18.171861 14744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.172132 14744 leader_election.cc:288] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e77e548c4d649fe94d198e6d68dc10e (127.1.50.126:46781), 0ba0e5bc0486455eb18435a3fb0b38e0 (127.1.50.125:45321)
I20260927 03:56:18.172740 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43e476f03a2946488067649caa24f3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e77e548c4d649fe94d198e6d68dc10e" is_pre_election: true
I20260927 03:56:18.172909 14555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e476f03a2946488067649caa24f3eb in term 0.
I20260927 03:56:18.173110 14682 leader_election.cc:302] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e77e548c4d649fe94d198e6d68dc10e, 43e476f03a2946488067649caa24f3eb; no voters:
I20260927 03:56:18.173287 14744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:56:18.173341 14744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:56:18.173369 14744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:18.174319 14744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.174464 14744 leader_election.cc:288] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [CANDIDATE]: Term 1 election: Requested vote from peers 1e77e548c4d649fe94d198e6d68dc10e (127.1.50.126:46781), 0ba0e5bc0486455eb18435a3fb0b38e0 (127.1.50.125:45321)
I20260927 03:56:18.174743 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43e476f03a2946488067649caa24f3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e77e548c4d649fe94d198e6d68dc10e"
I20260927 03:56:18.174836 14555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:18.174877 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43e476f03a2946488067649caa24f3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0"
I20260927 03:56:18.175015 14629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:18.175776 14555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e476f03a2946488067649caa24f3eb in term 1.
I20260927 03:56:18.175990 14629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e476f03a2946488067649caa24f3eb in term 1.
I20260927 03:56:18.176118 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43e476f03a2946488067649caa24f3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" is_pre_election: true
I20260927 03:56:18.176192 14630 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 43e476f03a2946488067649caa24f3eb in term 1. Re-sending same reply.
I20260927 03:56:18.176313 14682 leader_election.cc:302] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e77e548c4d649fe94d198e6d68dc10e, 43e476f03a2946488067649caa24f3eb; no voters:
I20260927 03:56:18.176512 14744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:18.176681 14744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [term 1 LEADER]: Becoming Leader. State: Replica: 43e476f03a2946488067649caa24f3eb, State: Running, Role: LEADER
I20260927 03:56:18.176783 14744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } }
I20260927 03:56:18.177217 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43e476f03a2946488067649caa24f3eb. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.177289 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:18.177492 14787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:18.177700 14787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:18.179186 14630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [term 1 FOLLOWER]: Refusing update from remote peer 43e476f03a2946488067649caa24f3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:18.179598 14555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [term 1 FOLLOWER]: Refusing update from remote peer 43e476f03a2946488067649caa24f3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:18.179630 14785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:18.179982 14785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:18.181102 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43e476f03a2946488067649caa24f3eb. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.181186 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:56:18.181223 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43e476f03a2946488067649caa24f3eb. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.181315 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:56:18.182574 14785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.182652 14785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:18.182734 14785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.182776 14785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:18.183244 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.183323 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:56:18.183372 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43e476f03a2946488067649caa24f3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e77e548c4d649fe94d198e6d68dc10e" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 46781 } } peers { permanent_uuid: "0ba0e5bc0486455eb18435a3fb0b38e0" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 45321 } } peers { permanent_uuid: "43e476f03a2946488067649caa24f3eb" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42393 } } }
I20260927 03:56:18.183442 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:56:18.183545 14790 mvcc.cc:205] Tried to move back new op lower bound from 7333811725019774976 to 7333811725012791296. Current Snapshot: MvccSnapshot[applied={T|T < 7333811725019774976}]
I20260927 03:56:18.183635 14789 mvcc.cc:205] Tried to move back new op lower bound from 7333811725019774976 to 7333811725012791296. Current Snapshot: MvccSnapshot[applied={T|T < 7333811725019774976}]
I20260927 03:56:18.184624 14787 catalog_manager.cc:1347] Generated new cluster ID: 63776fd69492436f8a8b2aa84626495d
I20260927 03:56:18.184671 14787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:18.196606 14787 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:56:18.197173 14787 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:18.206794 14787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43e476f03a2946488067649caa24f3eb: Generated new TSK 0
I20260927 03:56:18.207038 14787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:56:19.027472 14769 catalog_manager.cc:1259] Loaded cluster ID: 63776fd69492436f8a8b2aa84626495d
I20260927 03:56:19.027549 14769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: loading cluster ID for follower catalog manager: success
I20260927 03:56:19.028348 14769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: acquiring CA information for follower catalog manager: success
I20260927 03:56:19.028625 14769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ba0e5bc0486455eb18435a3fb0b38e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:56:19.049916 14783 catalog_manager.cc:1259] Loaded cluster ID: 63776fd69492436f8a8b2aa84626495d
I20260927 03:56:19.049990 14783 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: loading cluster ID for follower catalog manager: success
I20260927 03:56:19.050858 14783 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: acquiring CA information for follower catalog manager: success
I20260927 03:56:19.051146 14783 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e77e548c4d649fe94d198e6d68dc10e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:56:19.254352 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14513
I20260927 03:56:19.259589 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14588
I20260927 03:56:19.265077 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14663
2026-09-27T03:56:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2919 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:56:19 dist-test-slave-96fg krb5kdc[14802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:56:19 dist-test-slave-96fg krb5kdc[14802](info): set up 2 sockets
Sep 27 03:56:19 dist-test-slave-96fg krb5kdc[14802](info): commencing operation
krb5kdc: starting...
W20260927 03:56:20.238013 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:56:20.361783 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35767
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:20.445979 14815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:20.446230 14815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:20.446264 14815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:20.447553 14815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:20.447602 14815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:20.447620 14815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:20.447636 14815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:20.447654 14815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:20.449290 14815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:35767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:20.449540 14815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:20.449770 14815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:20.454594 14821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:20.454610 14820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:20.454872 14815 server_base.cc:1034] running on GCE node
W20260927 03:56:20.454610 14823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:20.455142 14815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:20.455451 14815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:20.456590 14815 hybrid_clock.cc:648] HybridClock initialized: now 1790481380456579 us; error 32 us; skew 500 ppm
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481380, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:20.459082 14815 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:20.459554 14815 webserver.cc:460] Webserver started at http://127.1.50.126:45633/ using document root <none> and password file <none>
I20260927 03:56:20.459767 14815 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:20.459810 14815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:20.459929 14815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:20.460978 14815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "b6c26c9f8ec64106b242389617270230"
format_stamp: "Formatted at 2026-09-27 03:56:20 on dist-test-slave-96fg"
server_key: "2a5495b0408349fd85b93cc2981f647f"
server_key_iv: "fabe77db84aa96bc68b25784b1cce549"
server_key_version: "encryptionkey@0"
I20260927 03:56:20.461303 14815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "b6c26c9f8ec64106b242389617270230"
format_stamp: "Formatted at 2026-09-27 03:56:20 on dist-test-slave-96fg"
server_key: "2a5495b0408349fd85b93cc2981f647f"
server_key_iv: "fabe77db84aa96bc68b25784b1cce549"
server_key_version: "encryptionkey@0"
I20260927 03:56:20.462594 14815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:56:20.463325 14830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:20.463528 14815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:20.463616 14815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "b6c26c9f8ec64106b242389617270230"
format_stamp: "Formatted at 2026-09-27 03:56:20 on dist-test-slave-96fg"
server_key: "2a5495b0408349fd85b93cc2981f647f"
server_key_iv: "fabe77db84aa96bc68b25784b1cce549"
server_key_version: "encryptionkey@0"
I20260927 03:56:20.463676 14815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:20.480193 14815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:20.482514 14815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:20.482685 14815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:20.486557 14815 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:35767
I20260927 03:56:20.486599 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:35767 every 8 connection(s)
I20260927 03:56:20.487010 14815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:20.487854 14883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:20.490242 14883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230: Bootstrap starting.
I20260927 03:56:20.490895 14883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:20.491127 14883 log.cc:824] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:20.491909 14883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230: No bootstrap required, opened a new log
I20260927 03:56:20.493460 14883 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:56:20.493610 14883 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:20.493641 14883 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c26c9f8ec64106b242389617270230, State: Initialized, Role: FOLLOWER
I20260927 03:56:20.493762 14883 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:56:20.493851 14883 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:20.493881 14883 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:20.493916 14883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:20.494441 14883 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:56:20.494529 14883 leader_election.cc:302] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c26c9f8ec64106b242389617270230; no voters:
I20260927 03:56:20.494659 14883 leader_election.cc:288] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:56:20.494720 14886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:20.494845 14886 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [term 1 LEADER]: Becoming Leader. State: Replica: b6c26c9f8ec64106b242389617270230, State: Running, Role: LEADER
I20260927 03:56:20.494926 14883 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:20.494989 14886 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } }
I20260927 03:56:20.495370 14888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c26c9f8ec64106b242389617270230. Latest consensus state: current_term: 1 leader_uuid: "b6c26c9f8ec64106b242389617270230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } } }
I20260927 03:56:20.495487 14887 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c26c9f8ec64106b242389617270230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c26c9f8ec64106b242389617270230" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 35767 } } }
I20260927 03:56:20.495477 14888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:20.495828 14887 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:20.495862 14895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:20.496404 14895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:20.496863 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14815
I20260927 03:56:20.496959 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:56:20.497149 1225 external_mini_cluster.cc:1428] Setting key 007ebf9a6aa963d7af9316e8b2354e55
I20260927 03:56:20.497856 14895 catalog_manager.cc:1347] Generated new cluster ID: ddd59c47d07c4b1daae2ae8516944a4f
I20260927 03:56:20.497901 14895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:20.503696 14895 catalog_manager.cc:1370] Generated new certificate authority record
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:20.504396 14895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:20.508352 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:20.497607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.1.50.126:35767)
0927 03:56:20.497969 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:20.497979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:20.498010 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:56:20.498145 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:20.498151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:20.498203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:20.498331 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:20.499062 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:20.500425 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:20.501506 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:20.502148 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:20.502245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:20.504793 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:20.504798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:20.504798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:20.504811 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:20.506755 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:20.507119 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:20.507121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:20.507122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:20.507238 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:20.507412 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:20.507413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:20.507413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:20.507699 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:20.507847 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:20.507929 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:56:20.508089 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":58,"threads_started":1}
I20260927 03:56:20.509296 14895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c26c9f8ec64106b242389617270230: Generated new TSK 0
I20260927 03:56:20.509485 14895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:56:20.509814 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:56:20.512941 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14815
2026-09-27T03:56:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1244 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14910](info): set up 2 sockets
Sep 27 03:56:20 dist-test-slave-96fg krb5kdc[14910](info): commencing operation
krb5kdc: starting...
W20260927 03:56:21.477839 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:56:21 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:56:21.601086 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:21.684099 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:21.684340 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:21.684365 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:21.685599 14923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:21.685645 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:21.685658 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:21.685670 14923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:21.685681 14923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:21.687575 14923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:21.687883 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:21.688107 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:21.693336 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:21.693459 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:21.693671 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:21.693888 14923 server_base.cc:1034] running on GCE node
I20260927 03:56:21.694176 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:21.694542 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:21.695696 14923 hybrid_clock.cc:648] HybridClock initialized: now 1790481381695662 us; error 50 us; skew 500 ppm
Sep 27 03:56:21 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:21.704973 14923 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:21.705487 14923 webserver.cc:460] Webserver started at http://127.1.50.126:43905/ using document root <none> and password file <none>
I20260927 03:56:21.705696 14923 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:21.705744 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:21.705935 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:21.707043 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "656f6b0a978f40e6bdc44735a96b1f46"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "44b434ae1a2f144c98afa6e959aa40d6"
server_key_iv: "12932b8f02fe6f7ada8b6b6d3dc08403"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.707369 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "656f6b0a978f40e6bdc44735a96b1f46"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "44b434ae1a2f144c98afa6e959aa40d6"
server_key_iv: "12932b8f02fe6f7ada8b6b6d3dc08403"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.708693 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:56:21.709729 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:21.709941 14923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:21.710062 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "656f6b0a978f40e6bdc44735a96b1f46"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "44b434ae1a2f144c98afa6e959aa40d6"
server_key_iv: "12932b8f02fe6f7ada8b6b6d3dc08403"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.710134 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:21.720202 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:21.722426 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:21.722564 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:21.726828 14923 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34767
I20260927 03:56:21.726877 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34767 every 8 connection(s)
I20260927 03:56:21.727237 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:21.728363 14991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:21.730839 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap starting.
I20260927 03:56:21.731381 14991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:21.731606 14991 log.cc:824] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:21.732362 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: No bootstrap required, opened a new log
I20260927 03:56:21.734054 14991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:21.734225 14991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:21.734251 14991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Initialized, Role: FOLLOWER
I20260927 03:56:21.734360 14991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:21.734411 14991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:21.734436 14991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:21.734467 14991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:21.735301 14991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:21.735397 14991 leader_election.cc:302] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656f6b0a978f40e6bdc44735a96b1f46; no voters:
I20260927 03:56:21.735533 14991 leader_election.cc:288] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:56:21.735599 14994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:21.735739 14994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 LEADER]: Becoming Leader. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Running, Role: LEADER
I20260927 03:56:21.735809 14991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:21.735818 14994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:21.736279 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656f6b0a978f40e6bdc44735a96b1f46. Latest consensus state: current_term: 1 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:21.736320 14995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:21.736382 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:21.736398 14995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:21.736476 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 14923
I20260927 03:56:21.736578 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:56:21.736609 14999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:21.736776 1225 external_mini_cluster.cc:1428] Setting key 6e9e1e8430053e66b2858cc373806afc
I20260927 03:56:21.737105 14999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:21.738806 14999 catalog_manager.cc:1347] Generated new cluster ID: 2e2205516bac4addb4032055cfe3c1d4
I20260927 03:56:21.738853 14999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:21.742928 14999 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:56:21.743573 14999 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 03:56:21 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:21.749053 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:21.737269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.1.50.126:34767)
0927 03:56:21.737469 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:21.737478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:21.737760 (+ 282us) server_negotiation.cc:407] Connection header received
0927 03:56:21.737884 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:21.737892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:21.737950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:21.738106 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:21.739021 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.740369 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.741490 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.742197 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.742303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:21.745252 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:21.745258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:21.745258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:21.745273 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:21.747358 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:21.747758 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:21.747762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:21.747763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:21.747883 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:21.748068 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:21.748069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:21.748069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:21.748402 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:21.748564 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:21.748639 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:56:21.748809 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260927 03:56:21.752995 14999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Generated new TSK 0
I20260927 03:56:21.753177 14999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:56:21.780993 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:21.865465 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:21.865669 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:21.865692 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:21.866922 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:21.866967 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:21.866993 15017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:21.868479 15017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:21.868700 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:21.868929 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:21.873795 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:21.873827 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:21.873891 15017 server_base.cc:1034] running on GCE node
W20260927 03:56:21.873986 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:21.874207 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:21.874423 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:21.875581 15017 hybrid_clock.cc:648] HybridClock initialized: now 1790481381875569 us; error 27 us; skew 500 ppm
Sep 27 03:56:21 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:21.878083 15017 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:21.878538 15017 webserver.cc:460] Webserver started at http://127.1.50.65:38815/ using document root <none> and password file <none>
I20260927 03:56:21.878729 15017 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:21.878768 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:21.878861 15017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:21.879923 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "90b1952eee954485a75edef87a2844d9"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "99aa801cc9a56dd6e94c5f4d659942fa"
server_key_iv: "e1045a978170bdf839f040f49dc9d1c8"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.880211 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "90b1952eee954485a75edef87a2844d9"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "99aa801cc9a56dd6e94c5f4d659942fa"
server_key_iv: "e1045a978170bdf839f040f49dc9d1c8"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.881407 15017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:56:21.882158 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:21.882344 15017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:21.882431 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "90b1952eee954485a75edef87a2844d9"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "99aa801cc9a56dd6e94c5f4d659942fa"
server_key_iv: "e1045a978170bdf839f040f49dc9d1c8"
server_key_version: "encryptionkey@0"
I20260927 03:56:21.882501 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:21.902915 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:21.905066 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:21.905203 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:21.905429 15017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:21.905731 15017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:21.905764 15017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:21.905794 15017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:21.905817 15017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:21.911819 15017 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:39185
I20260927 03:56:21.911908 15145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:39185 every 8 connection(s)
I20260927 03:56:21.912181 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:56:21.916111 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15017
I20260927 03:56:21.916219 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:56:21.916409 1225 external_mini_cluster.cc:1428] Setting key b380aa36e38f47fcc36675674fb368d0
Sep 27 03:56:21 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:21.921975 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:21.912917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:40887 (local address 127.1.50.126:34767)
0927 03:56:21.912983 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:21.912985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:21.913199 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:56:21.913711 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:21.913712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:21.913724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:21.913774 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:21.914772 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.915288 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.916301 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.916523 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.916582 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:21.918952 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:21.918954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:21.918954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:21.918962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:21.920306 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:21.920864 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:21.920866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:21.920866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:21.920942 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:21.921213 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:21.921214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:21.921214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:21.921382 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:21.921486 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:21.921758 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 03:56:21.921869 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:56:21.922062 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:21.912927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:40887)
0927 03:56:21.913083 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:21.913120 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:21.913614 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:21.913878 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:21.913883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:21.914487 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:21.914678 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:21.914683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.915369 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.915370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:21.916190 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:21.916193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:21.916591 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:21.916592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:21.916733 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:21.917307 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:21.917316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:21.918853 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:21.920404 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:21.920413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:21.920416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:21.920714 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:21.921024 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:21.921024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:21.921025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:21.921146 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:21.921463 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:21.921466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:21.921587 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:56:21.921667 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:56:21.921860 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260927 03:56:21.922572 15146 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:21.922667 15146 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:21.922825 15146 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:21.923297 14955 ts_manager.cc:194] Registered new tserver with Master: 90b1952eee954485a75edef87a2844d9 (127.1.50.65:39185)
I20260927 03:56:21.923936 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:40887
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:56:21.946517 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:22.027434 15151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.027653 15151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.027676 15151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.028887 15151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.028937 15151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.028986 15151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:22.030520 15151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.030761 15151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.030985 15151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:56:22.036021 15151 server_base.cc:1034] running on GCE node
W20260927 03:56:22.035988 15156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.036032 15159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.035988 15157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.036399 15151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.036664 15151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.037846 15151 hybrid_clock.cc:648] HybridClock initialized: now 1790481382037824 us; error 35 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.040422 15151 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:22.040885 15151 webserver.cc:460] Webserver started at http://127.1.50.66:46599/ using document root <none> and password file <none>
I20260927 03:56:22.041107 15151 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.041158 15151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.041287 15151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:22.042446 15151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "383dba6283134ef6a05cf94263123de4"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "0da6b5b54827add809727e74fb46285e"
server_key_iv: "c78dd7f11878c40a5a2c863bb608db98"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.042788 15151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "383dba6283134ef6a05cf94263123de4"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "0da6b5b54827add809727e74fb46285e"
server_key_iv: "c78dd7f11878c40a5a2c863bb608db98"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.044066 15151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.044734 15166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.044872 15151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:56:22.044952 15151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "383dba6283134ef6a05cf94263123de4"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "0da6b5b54827add809727e74fb46285e"
server_key_iv: "c78dd7f11878c40a5a2c863bb608db98"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.045013 15151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.062994 15151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.065153 15151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.065308 15151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.065531 15151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:22.065846 15151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:22.065881 15151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.065909 15151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:22.065932 15151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.071692 15151 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:38633
I20260927 03:56:22.071748 15279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:38633 every 8 connection(s)
I20260927 03:56:22.072059 15151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:22.081097 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.072773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:44357 (local address 127.1.50.126:34767)
0927 03:56:22.072841 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.072843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.073087 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:56:22.073624 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.073625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.073636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.073693 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.074754 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.075281 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.076332 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.076533 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.076596 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.078807 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.078809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.078809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.078816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.079937 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.080302 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.080302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.080303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.080365 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.080550 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.080550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.080550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.080647 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.080692 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.080970 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.081028 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:56:22.081314 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.072757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:44357)
0927 03:56:22.072970 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.073005 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.073505 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.073758 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.073761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.074416 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:22.074632 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.074637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.075399 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.075400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.076235 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.076238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.076618 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.076619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.076736 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.077251 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:22.077259 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:22.078708 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:22.080004 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.080011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.080013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.080228 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.080407 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.080407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.080407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.080505 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.080716 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:22.080722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:22.080857 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.080950 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.081078 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260927 03:56:22.081671 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15151
I20260927 03:56:22.081770 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:56:22.081944 15280 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:22.081996 1225 external_mini_cluster.cc:1428] Setting key 278c9f9f620d87f22358545ed16c0274
I20260927 03:56:22.082075 15280 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:22.082273 15280 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:22.082669 14955 ts_manager.cc:194] Registered new tserver with Master: 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633)
I20260927 03:56:22.083045 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:44357
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:56:22.110359 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:22.191672 15285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.191898 15285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.191924 15285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.193073 15285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.193111 15285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.193137 15285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:22.194597 15285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.194805 15285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.195019 15285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:22.199961 15291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.199975 15293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.200105 15285 server_base.cc:1034] running on GCE node
W20260927 03:56:22.200016 15290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.200385 15285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.200628 15285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.201830 15285 hybrid_clock.cc:648] HybridClock initialized: now 1790481382201807 us; error 34 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.204579 15285 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:22.205076 15285 webserver.cc:460] Webserver started at http://127.1.50.67:33811/ using document root <none> and password file <none>
I20260927 03:56:22.205317 15285 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.205361 15285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.205453 15285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:22.206518 15285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "39c2606f77f64a77be8c795307ea62bb"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "672aa4f25f1529eec63ade00b328e023"
server_key_iv: "51335b692ff3406df02816113f0c82b3"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.206809 15285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "39c2606f77f64a77be8c795307ea62bb"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "672aa4f25f1529eec63ade00b328e023"
server_key_iv: "51335b692ff3406df02816113f0c82b3"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.208225 15285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:56:22.208989 15300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.209185 15285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.209278 15285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "39c2606f77f64a77be8c795307ea62bb"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "672aa4f25f1529eec63ade00b328e023"
server_key_iv: "51335b692ff3406df02816113f0c82b3"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.209339 15285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.224673 15285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.226711 15285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.226827 15285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.227008 15285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:22.227257 15285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:22.227286 15285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.227319 15285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:22.227342 15285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.232762 15285 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40379
I20260927 03:56:22.232805 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40379 every 8 connection(s)
I20260927 03:56:22.233109 15285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:56:22.235636 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15285
I20260927 03:56:22.235742 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:56:22.235925 1225 external_mini_cluster.cc:1428] Setting key 4d008ed8753f03c4ec10f42a9902ca09
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:22.242939 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.233867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:54349 (local address 127.1.50.126:34767)
0927 03:56:22.233945 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.233947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.234048 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:56:22.234595 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.234596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.234606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.234656 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.235692 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.236341 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.237629 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.237832 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.237888 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.240123 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.240124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.240124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.240132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.241455 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.241850 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.241850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.241851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.241901 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.242240 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.242241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.242242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.242363 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.242423 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.242802 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.242870 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:56:22.243135 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.233771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:54349)
0927 03:56:22.233920 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.233957 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.234503 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.234709 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.234712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.235288 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:22.235546 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.235553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.236472 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.236473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.237555 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.237558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.237898 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.237898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.237999 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.238568 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:22.238577 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:22.240025 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:22.241541 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.241548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.241550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.241781 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.241985 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.241986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.241987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.242132 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.242451 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:22.242457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:22.242620 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.242769 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.242924 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260927 03:56:22.243589 15414 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:22.243687 15414 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:22.243865 15414 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:22.244264 14955 ts_manager.cc:194] Registered new tserver with Master: 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:22.244606 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:54349
I20260927 03:56:22.247561 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:56:22.254793 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.248886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.1.50.126:34767)
0927 03:56:22.248941 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.248943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.249135 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:56:22.249202 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.249202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.249210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.249250 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.249824 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.250359 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.251057 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.251233 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.251281 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.252362 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.252363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.252364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.252369 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.253669 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.253997 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.253998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.253999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.254083 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.254268 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.254268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.254268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.254404 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.254477 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.254592 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.254664 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:56:22.256428 14955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:56:22.258065 14955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:56:22.264815 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.260578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38633 (local address 127.0.0.1:37358)
0927 03:56:22.260847 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.260889 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.260958 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.261485 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.261486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.261488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.261542 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.261544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.262443 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.262444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.263657 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.263660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.264505 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.264506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.264585 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.264599 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.264666 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.264721 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260927 03:56:22.264931 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.260582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40379 (local address 127.0.0.1:32892)
0927 03:56:22.261178 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.261186 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.261220 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.261443 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.261443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.261445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.261485 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.261486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.262824 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.262825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.263834 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.263836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.264641 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.264643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.264715 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.264720 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.264764 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.264808 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260927 03:56:22.264981 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.261024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.1.50.67:40379)
0927 03:56:22.261206 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.261210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.261219 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:22.261281 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.261282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.261297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.261361 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.261552 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.262715 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.263913 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.264574 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.264627 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.264683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.264851 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.264891 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260927 03:56:22.265156 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.260845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.1.50.66:38633)
0927 03:56:22.261226 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.261231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.261240 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:22.261295 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.261314 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.261330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.261409 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.261614 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.262357 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.263745 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.264419 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.264497 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.264559 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.265031 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.265080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":83,"threads_started":1}
I20260927 03:56:22.265676 15348 tablet_service.cc:1467] Processing CreateTablet for tablet 19507d6d354844a2b5a572d1e8fa632f (DEFAULT_TABLE table=test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.265776 15214 tablet_service.cc:1467] Processing CreateTablet for tablet 19507d6d354844a2b5a572d1e8fa632f (DEFAULT_TABLE table=test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.266093 15348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19507d6d354844a2b5a572d1e8fa632f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.266093 15214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19507d6d354844a2b5a572d1e8fa632f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.266506 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.260578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39185 (local address 127.0.0.1:50042)
0927 03:56:22.261049 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.261058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.261106 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.261382 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.261385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.261388 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.261473 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.261475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.263912 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.263912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.264681 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.264683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.266302 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.266303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.266351 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.266353 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.266398 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.266439 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260927 03:56:22.266533 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.260828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.1.50.65:39185)
0927 03:56:22.260974 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.260979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.261107 (+ 128us) server_negotiation.cc:407] Connection header received
0927 03:56:22.261177 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.261183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.261199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.261281 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.261553 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.262724 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.264739 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.266163 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.266223 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.266275 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.266420 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.266457 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260927 03:56:22.269321 15431 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Bootstrap starting.
I20260927 03:56:22.269691 15080 tablet_service.cc:1467] Processing CreateTablet for tablet 19507d6d354844a2b5a572d1e8fa632f (DEFAULT_TABLE table=test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.269977 15080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19507d6d354844a2b5a572d1e8fa632f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.270000 15431 tablet_bootstrap.cc:654] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.270483 15431 log.cc:824] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.270794 15432 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Bootstrap starting.
I20260927 03:56:22.271294 15432 tablet_bootstrap.cc:654] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.271705 15432 log.cc:824] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.272449 15432 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: No bootstrap required, opened a new log
I20260927 03:56:22.272459 15431 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: No bootstrap required, opened a new log
I20260927 03:56:22.272521 15431 ts_tablet_manager.cc:1397] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:56:22.272521 15432 ts_tablet_manager.cc:1397] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:56:22.272822 15435 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Bootstrap starting.
I20260927 03:56:22.273339 15435 tablet_bootstrap.cc:654] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.273768 15435 log.cc:824] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.274394 15435 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: No bootstrap required, opened a new log
I20260927 03:56:22.274452 15435 ts_tablet_manager.cc:1397] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:56:22.274454 15431 raft_consensus.cc:348] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.274454 15432 raft_consensus.cc:348] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.274637 15432 raft_consensus.cc:374] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.274637 15431 raft_consensus.cc:374] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.274680 15431 raft_consensus.cc:729] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c2606f77f64a77be8c795307ea62bb, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.274680 15432 raft_consensus.cc:729] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383dba6283134ef6a05cf94263123de4, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.274817 15431 consensus_queue.cc:260] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.274816 15432 consensus_queue.cc:260] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.275100 15431 ts_tablet_manager.cc:1428] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:56:22.275100 15432 ts_tablet_manager.cc:1428] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:56:22.275208 15414 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
I20260927 03:56:22.275249 15280 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
I20260927 03:56:22.276132 15435 raft_consensus.cc:348] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.276505 15435 raft_consensus.cc:374] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.276554 15435 raft_consensus.cc:729] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.276667 15435 consensus_queue.cc:260] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.276893 15435 ts_tablet_manager.cc:1428] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:56:22.276942 15146 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
W20260927 03:56:22.322683 15281 tablet.cc:2369] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:22.374402 15439 raft_consensus.cc:482] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:22.374537 15439 raft_consensus.cc:504] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.374905 15439 leader_election.cc:288] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:22.377966 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.375273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35785 (local address 127.1.50.66:38633)
0927 03:56:22.375318 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.375319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.375420 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:56:22.375470 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.375471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.375482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.375537 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.375675 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.376278 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377256 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.377649 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377700 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.377709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.377858 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.377897 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:56:22.377991 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.375150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38633 (local address 127.1.50.65:35785)
0927 03:56:22.375344 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.375354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.375417 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.375547 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.375548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.375549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.375611 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.375612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.376346 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.376347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.377168 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.377170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.377726 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.377796 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.377800 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.377845 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.377884 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":49,"thread_start_us":109,"threads_started":1}
I20260927 03:56:22.377987 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.375309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55005 (local address 127.1.50.67:40379)
0927 03:56:22.375377 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.375378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.375387 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:22.375417 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.375418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.375430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.375482 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.375643 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.376272 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377284 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.377684 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.377761 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.377894 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.377929 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:56:22.377993 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.375150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40379 (local address 127.1.50.65:55005)
0927 03:56:22.375231 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.375245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.375327 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.375492 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.375493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.375494 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.375578 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.375579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.376362 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.376363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.377166 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.377168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.377763 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.377764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.377827 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.377831 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.377883 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.377923 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":31}
I20260927 03:56:22.378527 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19507d6d354844a2b5a572d1e8fa632f" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383dba6283134ef6a05cf94263123de4" is_pre_election: true
I20260927 03:56:22.378527 15368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19507d6d354844a2b5a572d1e8fa632f" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2606f77f64a77be8c795307ea62bb" is_pre_election: true
I20260927 03:56:22.378665 15234 raft_consensus.cc:2454] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 0.
I20260927 03:56:22.378672 15368 raft_consensus.cc:2454] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 0.
I20260927 03:56:22.378873 15033 leader_election.cc:302] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c2606f77f64a77be8c795307ea62bb, 90b1952eee954485a75edef87a2844d9; no voters:
I20260927 03:56:22.379099 15439 raft_consensus.cc:2781] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:56:22.379148 15439 raft_consensus.cc:482] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:56:22.379163 15439 raft_consensus.cc:3037] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:22.379820 15439 raft_consensus.cc:504] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.379938 15439 leader_election.cc:288] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 election: Requested vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:22.380091 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19507d6d354844a2b5a572d1e8fa632f" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383dba6283134ef6a05cf94263123de4"
I20260927 03:56:22.380084 15368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19507d6d354844a2b5a572d1e8fa632f" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2606f77f64a77be8c795307ea62bb"
I20260927 03:56:22.380167 15234 raft_consensus.cc:3037] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:22.380167 15368 raft_consensus.cc:3037] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:22.381029 15368 raft_consensus.cc:2454] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 1.
I20260927 03:56:22.381029 15234 raft_consensus.cc:2454] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 1.
I20260927 03:56:22.381208 15033 leader_election.cc:302] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c2606f77f64a77be8c795307ea62bb, 90b1952eee954485a75edef87a2844d9; no voters:
I20260927 03:56:22.381284 15439 raft_consensus.cc:2781] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:22.381438 15439 raft_consensus.cc:686] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 LEADER]: Becoming Leader. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Running, Role: LEADER
I20260927 03:56:22.381517 15439 consensus_queue.cc:237] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.382385 14953 catalog_manager.cc:5495] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b1952eee954485a75edef87a2844d9 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "90b1952eee954485a75edef87a2844d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } health_report { overall_health: UNKNOWN } } }
I20260927 03:56:22.409580 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.407199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.1.50.65:39185)
0927 03:56:22.407254 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.407256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.407266 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:56:22.407367 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.407372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.407397 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.407465 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:56:22.407570 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.408065 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.408825 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.409056 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.409118 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.409245 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:56:22.409349 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:56:22.409410 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.409444 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.409465 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:56:22.411371 15368 raft_consensus.cc:1261] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Refusing update from remote peer 90b1952eee954485a75edef87a2844d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:22.411356 15234 raft_consensus.cc:1261] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Refusing update from remote peer 90b1952eee954485a75edef87a2844d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:22.411723 15441 consensus_queue.cc:1035] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:22.411872 15439 consensus_queue.cc:1035] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:56:22.412914 15147 tablet.cc:2369] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:22.414745 15444 mvcc.cc:205] Tried to move back new op lower bound from 7333811742354558976 to 7333811742235475968. Current Snapshot: MvccSnapshot[applied={T|T < 7333811742354558976 or (T in {7333811742354558976})}]
I20260927 03:56:22.414985 15446 mvcc.cc:205] Tried to move back new op lower bound from 7333811742354558976 to 7333811742235475968. Current Snapshot: MvccSnapshot[applied={T|T < 7333811742354558976}]
I20260927 03:56:22.415169 15445 mvcc.cc:205] Tried to move back new op lower bound from 7333811742354558976 to 7333811742235475968. Current Snapshot: MvccSnapshot[applied={T|T < 7333811742354558976 or (T in {7333811742354558976})}]
I20260927 03:56:22.416527 14955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:22.416606 14955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:22.417263 14955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Sending DeleteTablet for 3 replicas of tablet 19507d6d354844a2b5a572d1e8fa632f
I20260927 03:56:22.417559 15214 tablet_service.cc:1514] Processing DeleteTablet for tablet 19507d6d354844a2b5a572d1e8fa632f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:22 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:37358
I20260927 03:56:22.417601 15348 tablet_service.cc:1514] Processing DeleteTablet for tablet 19507d6d354844a2b5a572d1e8fa632f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:22 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32892
I20260927 03:56:22.417682 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15017
I20260927 03:56:22.417753 15080 tablet_service.cc:1514] Processing DeleteTablet for tablet 19507d6d354844a2b5a572d1e8fa632f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:22 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:50042
I20260927 03:56:22.418540 15454 tablet_replica.cc:330] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: stopping tablet replica
I20260927 03:56:22.418644 15454 raft_consensus.cc:2229] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:22.418744 15454 raft_consensus.cc:2258] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:22.418890 15455 tablet_replica.cc:330] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: stopping tablet replica
I20260927 03:56:22.418985 15455 raft_consensus.cc:2229] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:22.419078 15455 raft_consensus.cc:2258] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:22.419355 15454 ts_tablet_manager.cc:1905] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:22.419443 15455 ts_tablet_manager.cc:1905] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:22.420976 15455 ts_tablet_manager.cc:1918] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:22.421044 15455 log.cc:1196] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/19507d6d354844a2b5a572d1e8fa632f
I20260927 03:56:22.421080 15454 ts_tablet_manager.cc:1918] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:22.421154 15454 log.cc:1196] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/19507d6d354844a2b5a572d1e8fa632f
I20260927 03:56:22.421298 15455 ts_tablet_manager.cc:1939] T 19507d6d354844a2b5a572d1e8fa632f P 39c2606f77f64a77be8c795307ea62bb: Deleting consensus metadata
I20260927 03:56:22.421422 15454 ts_tablet_manager.cc:1939] T 19507d6d354844a2b5a572d1e8fa632f P 383dba6283134ef6a05cf94263123de4: Deleting consensus metadata
I20260927 03:56:22.421716 14939 catalog_manager.cc:4841] TS 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): tablet 19507d6d354844a2b5a572d1e8fa632f (table test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]) successfully deleted
I20260927 03:56:22.421898 14941 catalog_manager.cc:4841] TS 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): tablet 19507d6d354844a2b5a572d1e8fa632f (table test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]) successfully deleted
W20260927 03:56:22.423548 14942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:39185 (error 108)
I20260927 03:56:22.423818 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15151
I20260927 03:56:22.424005 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.423898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39185 (local address 127.0.0.1:50068)
0927 03:56:22.423945 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.423973 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 03:56:22.424098 14942 catalog_manager.cc:4584] TS 90b1952eee954485a75edef87a2844d9 (127.1.50.65:39185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19507d6d354844a2b5a572d1e8fa632f: Network error: Client connection negotiation failed: client connection to 127.1.50.65:39185: connect: Connection refused (error 111)
I20260927 03:56:22.428535 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15285
I20260927 03:56:22.432255 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 14923
I20260927 03:56:22.436700 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--webserver_interface=127.1.50.126
--webserver_port=43905
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:22.518350 15457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.518581 15457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.518606 15457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.519815 15457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:22.519868 15457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.519883 15457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.519897 15457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:22.519909 15457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:22.521528 15457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=43905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.521773 15457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.522006 15457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:22.530668 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.530717 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.530686 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.531277 15457 server_base.cc:1034] running on GCE node
I20260927 03:56:22.531566 15457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.531872 15457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.533022 15457 hybrid_clock.cc:648] HybridClock initialized: now 1790481382533009 us; error 24 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.535557 15457 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:22.536010 15457 webserver.cc:460] Webserver started at http://127.1.50.126:43905/ using document root <none> and password file <none>
I20260927 03:56:22.536228 15457 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.536289 15457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.537715 15457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.538455 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.538605 15457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.538692 15457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "656f6b0a978f40e6bdc44735a96b1f46"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "44b434ae1a2f144c98afa6e959aa40d6"
server_key_iv: "12932b8f02fe6f7ada8b6b6d3dc08403"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.539088 15457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.551710 15457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.553862 15457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.553997 15457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.557756 15457 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34767
I20260927 03:56:22.557837 15525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34767 every 8 connection(s)
I20260927 03:56:22.558200 15457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:22.559916 15526 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:56:22.560593 15526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap starting.
I20260927 03:56:22.561617 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15457
I20260927 03:56:22.561929 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:39185
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=38815
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:22.569963 15526 log.cc:824] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.571946 15526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:56:22.572255 15526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap complete.
I20260927 03:56:22.574512 15526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:22.574759 15526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.574918 15526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:22.574975 15526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:22.575021 15526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:22.575065 15526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:56:22.576154 15526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:22.576248 15526 leader_election.cc:302] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656f6b0a978f40e6bdc44735a96b1f46; no voters:
I20260927 03:56:22.576412 15526 leader_election.cc:288] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:56:22.576462 15530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:56:22.576565 15530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 LEADER]: Becoming Leader. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Running, Role: LEADER
I20260927 03:56:22.576644 15530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:22.576678 15526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:22.576911 15530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:22.576980 15530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:22.577086 15530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656f6b0a978f40e6bdc44735a96b1f46. Latest consensus state: current_term: 2 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:22.577139 15530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:22.577260 15534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:22.578120 15534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:22.585879 15534 catalog_manager.cc:1259] Loaded cluster ID: 2e2205516bac4addb4032055cfe3c1d4
I20260927 03:56:22.585958 15534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:22.586299 15534 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 03:56:22.591420 15534 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 03:56:22.594187 15534 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:22.594447 15534 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Loaded TSK: 0
I20260927 03:56:22.594669 15534 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:56:22.649400 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.649633 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.649668 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.650992 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.651048 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.651089 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:22.652690 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:39185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=38815
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.652956 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.653172 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:22.658345 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.658308 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.658574 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.659189 15528 server_base.cc:1034] running on GCE node
I20260927 03:56:22.659406 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.659653 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.660846 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790481382660819 us; error 43 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.663635 15528 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:22.664149 15528 webserver.cc:460] Webserver started at http://127.1.50.65:38815/ using document root <none> and password file <none>
I20260927 03:56:22.664374 15528 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.664440 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.665970 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.666756 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.666957 15528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:56:22.667050 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "90b1952eee954485a75edef87a2844d9"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "99aa801cc9a56dd6e94c5f4d659942fa"
server_key_iv: "e1045a978170bdf839f040f49dc9d1c8"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.667425 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.677865 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.680078 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.680218 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.680444 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:22.680864 15570 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:56:22.683028 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:56:22.683095 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:56:22.683135 15528 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:56:22.683765 15528 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:56:22.683797 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:56:22.683893 15570 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Bootstrap starting.
I20260927 03:56:22.690816 15528 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:39185
I20260927 03:56:22.691196 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:56:22.694743 15677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:39185 every 8 connection(s)
I20260927 03:56:22.695561 15570 log.cc:824] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.698145 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15528
I20260927 03:56:22.698460 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:38633
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=46599
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:22.702803 15570 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:56:22.703186 15570 tablet_bootstrap.cc:492] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Bootstrap complete.
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:22.704265 15570 ts_tablet_manager.cc:1397] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20260927 03:56:22.705782 15570 raft_consensus.cc:348] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.706287 15570 raft_consensus.cc:729] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.706475 15570 consensus_queue.cc:260] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } }
I20260927 03:56:22.706823 15570 ts_tablet_manager.cc:1428] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260927 03:56:22.707934 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.691832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:53791)
0927 03:56:22.694241 (+ 2409us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.694399 (+ 158us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.695276 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.696330 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.696336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.696720 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:22.696971 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.696978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.698680 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.698681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.699976 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.699980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.701309 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.701311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.701482 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.702144 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:22.702154 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:22.703605 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:22.705688 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.705715 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.705717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.706009 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.706485 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.706485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.706486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.706688 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.707208 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:22.707214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:22.707378 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.707507 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.707720 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2347,"thread_start_us":90,"threads_started":1}
I20260927 03:56:22.708588 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.691936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:53791 (local address 127.1.50.126:34767)
0927 03:56:22.694497 (+ 2561us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.694505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.694555 (+ 50us) server_negotiation.cc:407] Connection header received
0927 03:56:22.695620 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.695627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.695697 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.695856 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.697127 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.698530 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.700087 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.701022 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.701143 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.703720 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.703725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.703725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.703737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.705606 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.706133 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.706138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.706138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.706269 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.706755 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.706756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.706757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.707116 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.707199 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.707604 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.707822 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2503,"thread_start_us":72,"threads_started":1}
I20260927 03:56:22.709247 15678 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:22.709357 15678 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:22.709568 15678 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:22.710451 15490 ts_manager.cc:194] Registered new tserver with Master: 90b1952eee954485a75edef87a2844d9 (127.1.50.65:39185)
I20260927 03:56:22.711676 15490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:53791
I20260927 03:56:22.712744 15678 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
I20260927 03:56:22.716989 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.711396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.1.50.65:39185)
0927 03:56:22.712011 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.712017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.712289 (+ 272us) server_negotiation.cc:407] Connection header received
0927 03:56:22.712336 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.712339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.712552 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.712627 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.712894 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.714153 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.715735 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.716506 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.716583 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.716641 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.716861 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.716907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":75,"threads_started":1}
I20260927 03:56:22.717005 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.711144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39185 (local address 127.0.0.1:50082)
0927 03:56:22.712083 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.712119 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.712197 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.712703 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.712706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.712709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.712805 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.712807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.714260 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.714261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.715618 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.715622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.716600 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.716601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.716734 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.716750 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.716831 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.716907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":105,"threads_started":1}
I20260927 03:56:22.717777 15606 tablet_service.cc:1514] Processing DeleteTablet for tablet 19507d6d354844a2b5a572d1e8fa632f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:22 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:50082
I20260927 03:56:22.718488 15687 tablet_replica.cc:330] stopping tablet replica
I20260927 03:56:22.718602 15687 raft_consensus.cc:2229] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:22.718679 15687 raft_consensus.cc:2258] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:22.719007 15687 ts_tablet_manager.cc:1905] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:22.721521 15687 ts_tablet_manager.cc:1918] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:22.721616 15687 log.cc:1196] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/19507d6d354844a2b5a572d1e8fa632f
I20260927 03:56:22.721839 15687 ts_tablet_manager.cc:1939] T 19507d6d354844a2b5a572d1e8fa632f P 90b1952eee954485a75edef87a2844d9: Deleting consensus metadata
I20260927 03:56:22.722455 15477 catalog_manager.cc:4841] TS 90b1952eee954485a75edef87a2844d9 (127.1.50.65:39185): tablet 19507d6d354844a2b5a572d1e8fa632f (table test-table [id=46dbdd5c6d634d0eb3df8aa4e060db4f]) successfully deleted
W20260927 03:56:22.796641 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.796864 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.796890 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.798156 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.798209 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.798237 15683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:22.799819 15683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:38633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=46599
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.800048 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.800235 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:22.806175 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.806181 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.806308 15683 server_base.cc:1034] running on GCE node
W20260927 03:56:22.806175 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.806685 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.806943 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.808148 15683 hybrid_clock.cc:648] HybridClock initialized: now 1790481382808068 us; error 97 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.811401 15683 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:22.811919 15683 webserver.cc:460] Webserver started at http://127.1.50.66:46599/ using document root <none> and password file <none>
I20260927 03:56:22.812150 15683 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.812208 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.813659 15683 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.814347 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.814527 15683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.814608 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "383dba6283134ef6a05cf94263123de4"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "0da6b5b54827add809727e74fb46285e"
server_key_iv: "c78dd7f11878c40a5a2c863bb608db98"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.815014 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.831275 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.833328 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.833451 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.833666 15683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:22.833977 15683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:22.834009 15683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.834075 15683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:22.834101 15683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.839811 15683 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:38633
I20260927 03:56:22.839862 15815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:38633 every 8 connection(s)
I20260927 03:56:22.840206 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:56:22.843639 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15683
I20260927 03:56:22.843924 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:40379
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=33811
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:22.850114 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.840935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:37105 (local address 127.1.50.126:34767)
0927 03:56:22.841015 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.841017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.841204 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:56:22.841735 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.841736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.841746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.841803 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.842808 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.843302 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.844644 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.844874 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.844935 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.847138 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.847140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.847140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.847147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.848737 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.849179 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.849180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.849181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.849235 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.849458 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.849459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.849459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.849571 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.849627 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.849893 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.849990 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:56:22.850298 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.840912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:37105)
0927 03:56:22.841072 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.841110 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.841609 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.841873 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.841877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.842509 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:22.842710 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.842715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.843386 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.843387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.844550 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.844554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.844953 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.844955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.845102 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.845607 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:22.845614 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:22.847031 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:22.848823 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.848832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.848834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.849110 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.849290 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.849291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.849291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.849403 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.849627 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:22.849634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:22.849775 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.849865 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.850074 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260927 03:56:22.850718 15816 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:22.850805 15816 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:22.850978 15816 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:22.851369 15490 ts_manager.cc:194] Registered new tserver with Master: 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633)
I20260927 03:56:22.851902 15490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:37105
W20260927 03:56:22.926389 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:22.926611 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:22.926637 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:22.927840 15819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:22.927882 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:22.927908 15819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:22.929419 15819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:40379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=33811
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:22.929642 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:22.929883 15819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:22.934970 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.934963 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:22.934966 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:22.935402 15819 server_base.cc:1034] running on GCE node
I20260927 03:56:22.935642 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:22.935882 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:22.937036 15819 hybrid_clock.cc:648] HybridClock initialized: now 1790481382937015 us; error 40 us; skew 500 ppm
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:22.939833 15819 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:22.940459 15819 webserver.cc:460] Webserver started at http://127.1.50.67:33811/ using document root <none> and password file <none>
I20260927 03:56:22.940665 15819 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:22.940728 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:22.942163 15819 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.942833 15834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.942981 15819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:56:22.943060 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "39c2606f77f64a77be8c795307ea62bb"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "672aa4f25f1529eec63ade00b328e023"
server_key_iv: "51335b692ff3406df02816113f0c82b3"
server_key_version: "encryptionkey@0"
I20260927 03:56:22.943416 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:22.954654 15819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:22.956732 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:22.956883 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:22.957105 15819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:22.957389 15819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:22.957422 15819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.957451 15819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:22.957472 15819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.963052 15819 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40379
I20260927 03:56:22.963097 15947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40379 every 8 connection(s)
I20260927 03:56:22.963428 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:56:22.969120 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15819
Sep 27 03:56:22 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:22.972298 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.964133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:34599 (local address 127.1.50.126:34767)
0927 03:56:22.964197 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.964199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.964400 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:56:22.964884 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.964885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.964894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.964962 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.965929 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.966472 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.967423 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.967603 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.967643 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.969665 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.969666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.969667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.969673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.971018 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.971385 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.971387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.971387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.971450 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.971669 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.971670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.971670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.971796 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.971853 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.972133 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.972208 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:56:22.972528 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.964104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:34599)
0927 03:56:22.964271 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.964311 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.964796 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.965023 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.965027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.965621 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:22.965808 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.965813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.966536 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.966536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.967331 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.967333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.967654 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.967654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.967753 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.968179 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:22.968187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:22.969579 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:22.971081 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.971089 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.971091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.971297 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.971518 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:22.971519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:22.971519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:22.971617 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:22.971849 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:22.971854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:22.972005 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.972119 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.972277 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260927 03:56:22.972980 15948 heartbeater.cc:347] Connected to a master server at 127.1.50.126:34767
I20260927 03:56:22.973090 15948 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:22.973277 15948 heartbeater.cc:511] Master 127.1.50.126:34767 requested a full tablet report, sending...
I20260927 03:56:22.973676 15490 ts_manager.cc:194] Registered new tserver with Master: 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:22.974087 15490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:34599
I20260927 03:56:22.975446 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.969491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.1.50.126:34767)
0927 03:56:22.969608 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.969610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.969619 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:56:22.969685 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.969686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.969697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.969749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:22.970413 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.970937 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.971815 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.972113 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.972172 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.973404 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:22.973405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:22.973406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:22.973412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:22.974685 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.974965 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.974965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.974966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.975009 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:22.975115 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:22.975115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:22.975115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:22.975195 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:22.975305 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.975330 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.975387 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260927 03:56:22.976034 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:56:22.978744 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.976332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.1.50.126:34767)
0927 03:56:22.976381 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.976382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.976466 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:56:22.976519 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.976520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.976545 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.976595 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:56:22.976665 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.977214 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.977934 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.978181 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.978229 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.978393 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:56:22.978498 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:56:22.978573 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.978643 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.978666 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:56:22.979046 15490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:56:22.980634 15490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:56:22.982879 15606 tablet_service.cc:1467] Processing CreateTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d (DEFAULT_TABLE table=test-table [id=bd725e7a179a417b8a5bc0742f05ee47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.983090 15606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.984728 15570 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Bootstrap starting.
I20260927 03:56:22.985076 15570 tablet_bootstrap.cc:654] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.985535 15570 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: No bootstrap required, opened a new log
I20260927 03:56:22.985576 15570 ts_tablet_manager.cc:1397] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:22.985680 15570 raft_consensus.cc:348] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.985759 15570 raft_consensus.cc:374] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.985813 15570 raft_consensus.cc:729] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.985862 15570 consensus_queue.cc:260] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.985973 15570 ts_tablet_manager.cc:1428] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:22.987047 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.982522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40379 (local address 127.0.0.1:32898)
0927 03:56:22.982673 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.982681 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.982714 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.983184 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.983185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.983188 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.983232 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.983234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.984241 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.984243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.985564 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.985644 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.986775 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.986776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.986863 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.986867 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.986912 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.986961 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":15,"thread_start_us":99,"threads_started":1}
I20260927 03:56:22.987061 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.982807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.1.50.67:40379)
0927 03:56:22.982970 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.982974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.982985 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:22.983022 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.983024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.983038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.983111 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.983297 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.984157 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.985768 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.986589 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.986644 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.986711 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.986939 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.986982 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260927 03:56:22.987373 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.982493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38633 (local address 127.0.0.1:37360)
0927 03:56:22.982539 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:22.982550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:22.982584 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:22.983041 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:22.983042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:22.983044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:22.983095 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.983097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.984457 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.984458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.985523 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:22.985526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.987101 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.987102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:22.987168 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.987171 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:56:22.987258 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:56:22.987306 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:56:22.987396 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:22.982585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.1.50.66:38633)
0927 03:56:22.982857 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:22.982862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:22.982873 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:22.982915 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:22.982917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:22.982932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:22.983165 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:22.983318 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.984320 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.986095 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:22.987002 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:22.987098 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:22.987155 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:22.987276 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:56:22.987318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":77,"threads_started":1}
I20260927 03:56:22.988184 15882 tablet_service.cc:1467] Processing CreateTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d (DEFAULT_TABLE table=test-table [id=bd725e7a179a417b8a5bc0742f05ee47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.988200 15750 tablet_service.cc:1467] Processing CreateTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d (DEFAULT_TABLE table=test-table [id=bd725e7a179a417b8a5bc0742f05ee47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:22.988523 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.988524 15882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:22.991582 15958 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Bootstrap starting.
I20260927 03:56:22.992158 15958 tablet_bootstrap.cc:654] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.992570 15958 log.cc:824] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.992966 15959 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Bootstrap starting.
I20260927 03:56:22.993372 15958 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: No bootstrap required, opened a new log
I20260927 03:56:22.993535 15958 ts_tablet_manager.cc:1397] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:56:22.993772 15959 tablet_bootstrap.cc:654] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:22.994439 15959 log.cc:824] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:22.995306 15959 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: No bootstrap required, opened a new log
I20260927 03:56:22.995384 15959 ts_tablet_manager.cc:1397] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:56:22.995617 15958 raft_consensus.cc:348] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.995816 15958 raft_consensus.cc:374] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.995853 15958 raft_consensus.cc:729] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383dba6283134ef6a05cf94263123de4, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.995975 15958 consensus_queue.cc:260] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.996296 15958 ts_tablet_manager.cc:1428] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:56:22.996358 15816 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
I20260927 03:56:22.997259 15959 raft_consensus.cc:348] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.997423 15959 raft_consensus.cc:374] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:22.997465 15959 raft_consensus.cc:729] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c2606f77f64a77be8c795307ea62bb, State: Initialized, Role: FOLLOWER
I20260927 03:56:22.997575 15959 consensus_queue.cc:260] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:22.997814 15959 ts_tablet_manager.cc:1428] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:56:22.997946 15948 heartbeater.cc:503] Master 127.1.50.126:34767 was elected leader, sending a full tablet report...
I20260927 03:56:23.081755 15684 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:23.081945 15684 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:23.082302 15684 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:23.085295 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.082577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:46025 (local address 127.1.50.67:40379)
0927 03:56:23.082622 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:23.082625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:23.082850 (+ 225us) server_negotiation.cc:407] Connection header received
0927 03:56:23.082881 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:23.082883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:23.082896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:23.082943 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:23.083074 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.083557 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.084584 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.084978 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.085038 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:23.085048 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:23.085184 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:56:23.085226 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:56:23.085304 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.082622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:53027 (local address 127.1.50.66:38633)
0927 03:56:23.082665 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:23.082667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:23.082675 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:56:23.082701 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:23.082702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:23.082710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:23.082868 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:23.083024 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.083500 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.084589 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.084977 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.085047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:23.085058 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:23.085192 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:56:23.085233 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:56:23.085350 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.082510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38633 (local address 127.1.50.65:53027)
0927 03:56:23.082560 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.082572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:23.082616 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:23.082903 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:23.082904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:23.082907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:23.082958 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:23.082959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.083580 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.083581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:23.084471 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:23.084473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.085054 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.085056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:23.085123 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:23.085127 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:23.085182 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:56:23.085226 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:56:23.085481 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.082514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40379 (local address 127.1.50.65:46025)
0927 03:56:23.082721 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.082728 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:23.082825 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:23.082944 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:23.082944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:23.082946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:23.083020 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:23.083022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.083688 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.083689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:23.084500 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:23.084503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.085069 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.085070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:23.085125 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:23.085128 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:56:23.085173 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:56:23.085218 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":68,"thread_start_us":95,"threads_started":1}
I20260927 03:56:23.085829 15902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0a04bac0bc4f4d98cc3a61f40c1f2d" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2606f77f64a77be8c795307ea62bb" is_pre_election: true
I20260927 03:56:23.085825 15770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0a04bac0bc4f4d98cc3a61f40c1f2d" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383dba6283134ef6a05cf94263123de4" is_pre_election: true
I20260927 03:56:23.085988 15902 raft_consensus.cc:2454] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 0.
I20260927 03:56:23.085988 15770 raft_consensus.cc:2454] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 0.
I20260927 03:56:23.086277 15564 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c2606f77f64a77be8c795307ea62bb, 90b1952eee954485a75edef87a2844d9; no voters:
I20260927 03:56:23.086391 15684 raft_consensus.cc:2781] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:56:23.086450 15684 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:56:23.086470 15684 raft_consensus.cc:3037] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:23.087137 15684 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:23.087260 15684 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 election: Requested vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
I20260927 03:56:23.087481 15902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0a04bac0bc4f4d98cc3a61f40c1f2d" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2606f77f64a77be8c795307ea62bb"
I20260927 03:56:23.087508 15770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0a04bac0bc4f4d98cc3a61f40c1f2d" candidate_uuid: "90b1952eee954485a75edef87a2844d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383dba6283134ef6a05cf94263123de4"
I20260927 03:56:23.087575 15770 raft_consensus.cc:3037] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:23.087569 15902 raft_consensus.cc:3037] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:23.088430 15902 raft_consensus.cc:2454] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 1.
I20260927 03:56:23.088430 15770 raft_consensus.cc:2454] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b1952eee954485a75edef87a2844d9 in term 1.
I20260927 03:56:23.088603 15564 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c2606f77f64a77be8c795307ea62bb, 90b1952eee954485a75edef87a2844d9; no voters:
I20260927 03:56:23.088685 15684 raft_consensus.cc:2781] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:23.088829 15684 raft_consensus.cc:686] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 LEADER]: Becoming Leader. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Running, Role: LEADER
I20260927 03:56:23.088893 15684 consensus_queue.cc:237] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:23.089648 15490 catalog_manager.cc:5495] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b1952eee954485a75edef87a2844d9 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "90b1952eee954485a75edef87a2844d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } health_report { overall_health: HEALTHY } } }
W20260927 03:56:23.090680 15817 tablet.cc:2369] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:23.104333 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.102145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.1.50.65:39185)
0927 03:56:23.102204 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:23.102206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:23.102214 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:56:23.102257 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:23.102258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:23.102278 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:23.102343 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:56:23.102456 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.102922 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.103616 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:23.103817 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:23.103878 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:23.104028 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:56:23.104128 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:56:23.104188 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:23.104221 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:56:23.104242 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:56:23.105888 15902 raft_consensus.cc:1261] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Refusing update from remote peer 90b1952eee954485a75edef87a2844d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:23.105877 15770 raft_consensus.cc:1261] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Refusing update from remote peer 90b1952eee954485a75edef87a2844d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:23.106271 15965 consensus_queue.cc:1035] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:23.106387 15684 consensus_queue.cc:1035] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:23.108636 15967 mvcc.cc:205] Tried to move back new op lower bound from 7333811745199071232 to 7333811745132670976. Current Snapshot: MvccSnapshot[applied={T|T < 7333811745199071232}]
I20260927 03:56:23.109261 15968 mvcc.cc:205] Tried to move back new op lower bound from 7333811745199071232 to 7333811745132670976. Current Snapshot: MvccSnapshot[applied={T|T < 7333811745199071232}]
I20260927 03:56:23.111683 15489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:23.111788 15489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:23.112582 15489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Sending DeleteTablet for 3 replicas of tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:23.112895 15606 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:23 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:50082
I20260927 03:56:23.112998 15687 tablet_replica.cc:330] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: stopping tablet replica
I20260927 03:56:23.112963 15750 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:23 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:37360
I20260927 03:56:23.113051 15687 raft_consensus.cc:2229] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:56:23.113150 15687 raft_consensus.cc:2258] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:23.113258 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15528
I20260927 03:56:23.113241 15882 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:23 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:32898
I20260927 03:56:23.114591 15977 tablet_replica.cc:330] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: stopping tablet replica
I20260927 03:56:23.114701 15977 raft_consensus.cc:2229] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:23.114812 15977 raft_consensus.cc:2258] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:23.114960 15978 tablet_replica.cc:330] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: stopping tablet replica
I20260927 03:56:23.115057 15978 raft_consensus.cc:2229] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:23.115147 15978 raft_consensus.cc:2258] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:23.115218 15977 ts_tablet_manager.cc:1905] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:23.115622 15978 ts_tablet_manager.cc:1905] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:23.117406 15977 ts_tablet_manager.cc:1918] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:23.117393 15978 ts_tablet_manager.cc:1918] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:23.117482 15977 log.cc:1196] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:23.117540 15978 log.cc:1196] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:23.117755 15977 ts_tablet_manager.cc:1939] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 383dba6283134ef6a05cf94263123de4: Deleting consensus metadata
I20260927 03:56:23.117797 15978 ts_tablet_manager.cc:1939] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 39c2606f77f64a77be8c795307ea62bb: Deleting consensus metadata
I20260927 03:56:23.118196 15476 catalog_manager.cc:4841] TS 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d (table test-table [id=bd725e7a179a417b8a5bc0742f05ee47]) successfully deleted
I20260927 03:56:23.118237 15474 catalog_manager.cc:4841] TS 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d (table test-table [id=bd725e7a179a417b8a5bc0742f05ee47]) successfully deleted
W20260927 03:56:23.118517 15477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:39185 (error 108)
I20260927 03:56:23.118747 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15683
I20260927 03:56:23.119445 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.118867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:39185 (local address 127.0.0.1:50110)
0927 03:56:23.119377 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.119417 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
W20260927 03:56:23.119572 15477 catalog_manager.cc:4584] TS 90b1952eee954485a75edef87a2844d9 (127.1.50.65:39185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee0a04bac0bc4f4d98cc3a61f40c1f2d: Network error: Client connection negotiation failed: client connection to 127.1.50.65:39185: connect: Connection refused (error 111)
I20260927 03:56:23.123473 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15819
I20260927 03:56:23.128049 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 15457
I20260927 03:56:23.132838 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--webserver_interface=127.1.50.126
--webserver_port=43905
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:23.217365 15980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:23.217589 15980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:23.217612 15980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:23.218870 15980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:23.218922 15980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:23.218935 15980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:23.218946 15980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:23.218957 15980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:23.220512 15980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=43905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:23.220790 15980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:23.221020 15980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:23.225843 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.225769 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:23.225924 15980 server_base.cc:1034] running on GCE node
W20260927 03:56:23.225832 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:23.226251 15980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:23.226498 15980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:23.227627 15980 hybrid_clock.cc:648] HybridClock initialized: now 1790481383227616 us; error 26 us; skew 500 ppm
Sep 27 03:56:23 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:23.229997 15980 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:23.230415 15980 webserver.cc:460] Webserver started at http://127.1.50.126:43905/ using document root <none> and password file <none>
I20260927 03:56:23.230609 15980 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:23.230648 15980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:23.232074 15980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.232854 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.233057 15980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.233145 15980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "656f6b0a978f40e6bdc44735a96b1f46"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "44b434ae1a2f144c98afa6e959aa40d6"
server_key_iv: "12932b8f02fe6f7ada8b6b6d3dc08403"
server_key_version: "encryptionkey@0"
I20260927 03:56:23.233505 15980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:23.250167 15980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:23.252918 15980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:23.253079 15980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:23.257524 15980 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34767
I20260927 03:56:23.257613 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34767 every 8 connection(s)
I20260927 03:56:23.257957 15980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:23.258140 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 15980
I20260927 03:56:23.258433 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:39185
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=38815
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:23.259620 16048 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:56:23.260289 16048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap starting.
I20260927 03:56:23.268484 16048 log.cc:824] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:23.271128 16048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:56:23.271396 16048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46: Bootstrap complete.
I20260927 03:56:23.273522 16048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:23.273756 16048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Initialized, Role: FOLLOWER
I20260927 03:56:23.273917 16048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:23.273993 16048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:23.274062 16048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:23.274109 16048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:56:23.275079 16048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:23.275173 16048 leader_election.cc:302] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656f6b0a978f40e6bdc44735a96b1f46; no voters:
I20260927 03:56:23.275341 16048 leader_election.cc:288] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:56:23.275400 16052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:56:23.275545 16052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [term 3 LEADER]: Becoming Leader. State: Replica: 656f6b0a978f40e6bdc44735a96b1f46, State: Running, Role: LEADER
I20260927 03:56:23.275586 16048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:23.275643 16052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } }
I20260927 03:56:23.276019 16052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656f6b0a978f40e6bdc44735a96b1f46. Latest consensus state: current_term: 3 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:23.276098 16052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:23.275939 16053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "656f6b0a978f40e6bdc44735a96b1f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f6b0a978f40e6bdc44735a96b1f46" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34767 } } }
I20260927 03:56:23.276278 16053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f6b0a978f40e6bdc44735a96b1f46 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:23.276305 16060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:23.277110 16060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:56:23.277210 16060 catalog_manager.cc:1259] Loaded cluster ID: 2e2205516bac4addb4032055cfe3c1d4
I20260927 03:56:23.277241 16060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 03:56:23.277449 16060 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe2035939ad google::LogMessage::Fail() at ??:0
@ 0x7fe20359688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe2035934de google::LogMessage::Flush() at ??:0
@ 0x7fe203594a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe20b4b80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe20b4c18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe2054610d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe205456954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2062ba6db start_thread at ??:0
@ 0x7fe20209161f clone at ??:0
W20260927 03:56:23.372998 16050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:23.373306 16050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:23.373355 16050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:23.375301 16050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:23.375389 16050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:23.375435 16050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:23.377817 16050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:39185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=38815
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:23.378129 16050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:23.378407 16050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:23.383639 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.383708 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.383639 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:23.384007 16050 server_base.cc:1034] running on GCE node
I20260927 03:56:23.384191 16050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:23.384420 16050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:23.385597 16050 hybrid_clock.cc:648] HybridClock initialized: now 1790481383385580 us; error 31 us; skew 500 ppm
Sep 27 03:56:23 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:23.388262 16050 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:23.388723 16050 webserver.cc:460] Webserver started at http://127.1.50.65:38815/ using document root <none> and password file <none>
I20260927 03:56:23.388936 16050 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:23.388984 16050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:23.390480 16050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.391219 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.391440 16050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.391534 16050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "90b1952eee954485a75edef87a2844d9"
format_stamp: "Formatted at 2026-09-27 03:56:21 on dist-test-slave-96fg"
server_key: "99aa801cc9a56dd6e94c5f4d659942fa"
server_key_iv: "e1045a978170bdf839f040f49dc9d1c8"
server_key_version: "encryptionkey@0"
I20260927 03:56:23.391912 16050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:23.404596 16050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:23.406829 16050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:23.406975 16050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:23.407199 16050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:23.407639 16101 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:56:23.409719 16050 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:56:23.409839 16050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:56:23.409883 16050 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:56:23.410597 16050 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:56:23.410631 16050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:56:23.410735 16101 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Bootstrap starting.
I20260927 03:56:23.417130 16050 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:39185
I20260927 03:56:23.417279 16208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:39185 every 8 connection(s)
I20260927 03:56:23.417544 16050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:56:23.421635 16101 log.cc:824] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:23.423902 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16050
I20260927 03:56:23.424220 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:38633
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=46599
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:56:23.424239 16101 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:56:23.424486 16101 tablet_bootstrap.cc:492] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Bootstrap complete.
I20260927 03:56:23.425525 16101 ts_tablet_manager.cc:1397] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260927 03:56:23.426640 16101 raft_consensus.cc:348] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:23.426882 16101 raft_consensus.cc:729] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1952eee954485a75edef87a2844d9, State: Initialized, Role: FOLLOWER
I20260927 03:56:23.427013 16101 consensus_queue.cc:260] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:23.427285 16101 ts_tablet_manager.cc:1428] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.005s
W20260927 03:56:23.518443 16211 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:56:23.418261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:59463)
0927 03:56:23.422042 (+ 3781us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.422105 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:23.422799 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:23.518238 (+ 95439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3710,"thread_start_us":82,"threads_started":1}
W20260927 03:56:23.518680 16209 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:34767 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:34767: Client connection negotiation failed: client connection to 127.1.50.126:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:56:23.519068 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.518931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:58915)
0927 03:56:23.519001 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.519044 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:56:23.519446 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.519362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:59925)
0927 03:56:23.519404 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.519428 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:56:23.519546 16209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:56:23.536908 16213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:23.537118 16213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:23.537149 16213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:23.538426 16213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:23.538475 16213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:23.538511 16213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:23.539974 16213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:38633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=46599
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:23.540199 16213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:23.540408 16213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:23.545291 16222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.545305 16219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.545296 16220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:23.545452 16213 server_base.cc:1034] running on GCE node
I20260927 03:56:23.545760 16213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:23.545980 16213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:23.547164 16213 hybrid_clock.cc:648] HybridClock initialized: now 1790481383547151 us; error 39 us; skew 500 ppm
Sep 27 03:56:23 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:23.549703 16213 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:23.550176 16213 webserver.cc:460] Webserver started at http://127.1.50.66:46599/ using document root <none> and password file <none>
I20260927 03:56:23.550360 16213 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:23.550411 16213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:23.551992 16213 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:56:23.552613 16229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.552799 16213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.552889 16213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "383dba6283134ef6a05cf94263123de4"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "0da6b5b54827add809727e74fb46285e"
server_key_iv: "c78dd7f11878c40a5a2c863bb608db98"
server_key_version: "encryptionkey@0"
I20260927 03:56:23.553231 16213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:23.562006 16213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:23.564064 16213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:23.564177 16213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:23.564399 16213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:23.564674 16213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:23.564706 16213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.564728 16213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:23.564739 16213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.570140 16213 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:38633
I20260927 03:56:23.570214 16342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:38633 every 8 connection(s)
I20260927 03:56:23.570533 16213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:56:23.571587 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.571199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:58065)
0927 03:56:23.571363 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.571434 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
W20260927 03:56:23.571976 16343 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:34767 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:34767: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
I20260927 03:56:23.572252 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.572170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:35597)
0927 03:56:23.572209 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.572233 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:56:23.572459 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.572411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:42013)
0927 03:56:23.572431 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.572448 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 03:56:23.572538 16343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:56:23.580580 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16213
I20260927 03:56:23.580833 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:40379
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:34767
--webserver_port=33811
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:23.662127 16346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:23.662374 16346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:23.662401 16346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:23.663695 16346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:23.663758 16346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:23.663789 16346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:23.665410 16346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:40379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=33811
--tserver_master_addrs=127.1.50.126:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:23.665663 16346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:23.665891 16346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:23.671231 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.671231 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:23.671262 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:23.671468 16346 server_base.cc:1034] running on GCE node
I20260927 03:56:23.671680 16346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:23.671916 16346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:23.673082 16346 hybrid_clock.cc:648] HybridClock initialized: now 1790481383673054 us; error 47 us; skew 500 ppm
Sep 27 03:56:23 dist-test-slave-96fg krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:23.675660 16346 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:23.676131 16346 webserver.cc:460] Webserver started at http://127.1.50.67:33811/ using document root <none> and password file <none>
I20260927 03:56:23.676338 16346 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:23.676384 16346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:23.677838 16346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:23.678541 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.678791 16346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:56:23.678886 16346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "39c2606f77f64a77be8c795307ea62bb"
format_stamp: "Formatted at 2026-09-27 03:56:22 on dist-test-slave-96fg"
server_key: "672aa4f25f1529eec63ade00b328e023"
server_key_iv: "51335b692ff3406df02816113f0c82b3"
server_key_version: "encryptionkey@0"
I20260927 03:56:23.679265 16346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:23.695833 16346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:23.697937 16346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:23.698143 16346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:23.698374 16346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:23.698664 16346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:23.698697 16346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.698719 16346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:23.698747 16346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:23.704842 16346 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40379
I20260927 03:56:23.704905 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40379 every 8 connection(s)
I20260927 03:56:23.705256 16346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:56:23.705955 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16346
I20260927 03:56:23.706489 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.705967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:55351)
0927 03:56:23.706176 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.706287 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
W20260927 03:56:23.706936 16475 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:34767 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:34767: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
I20260927 03:56:23.707209 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.707120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:40041)
0927 03:56:23.707164 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.707192 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:56:23.707454 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:23.707402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:45089)
0927 03:56:23.707423 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:23.707442 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 03:56:23.707595 16475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:56:24.520375 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.520037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:43259)
0927 03:56:24.520223 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:24.520288 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260927 03:56:24.573443 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.573062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:38961)
0927 03:56:24.573285 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:24.573356 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260927 03:56:24.708216 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.707944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:49847)
0927 03:56:24.708097 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:24.708155 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260927 03:56:24.750826 16482 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:24.750984 16482 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:24.751322 16482 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
Sep 27 03:56:24 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.66@KRBTEST.COM
Sep 27 03:56:24 dist-test-slave-96fg krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.67@KRBTEST.COM
I20260927 03:56:24.765671 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.751793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:54489 (local address 127.1.50.67:40379)
0927 03:56:24.751961 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:24.751999 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:24.752040 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:56:24.752595 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:24.752598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:24.752661 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:24.752861 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:24.753805 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.755300 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.757118 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.757880 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.758013 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:24.761806 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:24.761812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:24.761813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:24.761831 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:24.763791 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764262 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:24.764266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:24.764266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:24.764409 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764698 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:24.764699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:24.764700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:24.764997 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:24.765102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:24.765414 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 03:56:24.765558 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260927 03:56:24.765671 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.751736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:33153 (local address 127.1.50.66:38633)
0927 03:56:24.751898 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:24.751909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:24.751944 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:56:24.752464 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:24.752467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:24.752518 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:24.752643 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:24.753753 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.755255 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.757116 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.757869 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.758012 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:24.761348 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:24.761356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:24.761356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:24.761377 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:24.763539 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764110 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:24.764114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:24.764115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:24.764261 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764559 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:24.764560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:24.764560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:24.764937 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:24.765013 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:24.765395 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 03:56:24.765550 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":55,"threads_started":1}
I20260927 03:56:24.765657 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.751584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38633 (local address 127.1.50.65:33153)
0927 03:56:24.751722 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:24.751734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:24.751778 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:24.752747 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:24.752751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:24.753330 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:24.753588 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:24.753598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.755454 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.755455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:24.756982 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:24.756987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.758015 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.758047 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:24.758241 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:24.758971 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:24.758985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:24.761139 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:24.763653 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:24.763665 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:24.763669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:24.764007 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:24.764332 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:24.764333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:24.764467 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:24.765018 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:24.765026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:24.765186 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 03:56:24.765287 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:56:24.765515 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":72,"thread_start_us":69,"threads_started":1}
I20260927 03:56:24.765865 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:24.751561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40379 (local address 127.1.50.65:54489)
0927 03:56:24.751631 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:24.751643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:24.751706 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:24.752877 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:24.752878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:24.753337 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:24.753591 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:24.753602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.755475 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.755476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:24.756982 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:24.756987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:24.758075 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:24.758076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:24.758241 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:24.759043 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:24.759048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:24.761599 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:24.763919 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:24.763920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:24.763921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:24.764152 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:24.764514 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:24.764515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:24.764515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:24.764607 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:24.765087 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:24.765088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:24.765200 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 03:56:24.765334 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 03:56:24.765615 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":175}
W20260927 03:56:24.771341 16097 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
W20260927 03:56:24.771518 16095 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:24.771556 16095 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1952eee954485a75edef87a2844d9; no voters: 383dba6283134ef6a05cf94263123de4, 39c2606f77f64a77be8c795307ea62bb
I20260927 03:56:24.771732 16482 raft_consensus.cc:2726] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:56:25.521405 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:25.521080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:58853)
0927 03:56:25.521286 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:25.521330 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260927 03:56:25.574440 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:25.574137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:45167)
0927 03:56:25.574307 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:25.574375 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260927 03:56:25.709129 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:25.708834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:42727)
0927 03:56:25.708994 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:25.709050 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260927 03:56:26.522444 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:26.522167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:50909)
0927 03:56:26.522329 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:26.522390 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260927 03:56:26.575351 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:26.575060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:36115)
0927 03:56:26.575240 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:26.575292 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260927 03:56:26.710106 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:26.709817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:45013)
0927 03:56:26.709976 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:26.710046 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260927 03:56:26.901937 16492 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:26.902107 16492 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:26.902312 16492 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
W20260927 03:56:26.902845 16097 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
W20260927 03:56:26.902964 16095 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:26.903060 16095 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1952eee954485a75edef87a2844d9; no voters: 383dba6283134ef6a05cf94263123de4, 39c2606f77f64a77be8c795307ea62bb
I20260927 03:56:26.903162 16492 raft_consensus.cc:2726] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:56:27.523334 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:27.523025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:54077)
0927 03:56:27.523214 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:27.523268 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260927 03:56:27.576253 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:27.575966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:36299)
0927 03:56:27.576139 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:27.576186 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260927 03:56:27.711187 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:27.710831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:57439)
0927 03:56:27.711062 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:27.711129 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260927 03:56:28.492801 16496 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:28.492944 16496 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:28.493175 16496 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
W20260927 03:56:28.493726 16095 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
W20260927 03:56:28.493854 16097 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:28.493914 16097 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1952eee954485a75edef87a2844d9; no voters: 383dba6283134ef6a05cf94263123de4, 39c2606f77f64a77be8c795307ea62bb
I20260927 03:56:28.494045 16496 raft_consensus.cc:2726] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:56:28.524348 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:28.524048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:51571)
0927 03:56:28.524238 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:28.524281 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260927 03:56:28.577183 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:28.576859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:55853)
0927 03:56:28.577059 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:28.577123 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260927 03:56:28.712149 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:28.711808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:50975)
0927 03:56:28.712033 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:28.712096 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
I20260927 03:56:29.525256 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:29.524945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:53299)
0927 03:56:29.525147 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:29.525205 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260927 03:56:29.578322 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:29.577851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:46563)
0927 03:56:29.578203 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:29.578256 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":85,"threads_started":1}
I20260927 03:56:29.713115 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:29.712801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:56431)
0927 03:56:29.713010 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:29.713068 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
I20260927 03:56:30.069722 16503 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:30.069855 16503 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:30.070195 16503 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
W20260927 03:56:30.070564 16097 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
W20260927 03:56:30.070720 16095 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:30.070803 16095 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1952eee954485a75edef87a2844d9; no voters: 383dba6283134ef6a05cf94263123de4, 39c2606f77f64a77be8c795307ea62bb
I20260927 03:56:30.070894 16503 raft_consensus.cc:2726] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:56:30.526384 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:30.526002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:43377)
0927 03:56:30.526256 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:30.526332 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":145,"threads_started":1}
I20260927 03:56:30.579396 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:30.579065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:55289)
0927 03:56:30.579262 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:30.579337 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
I20260927 03:56:30.714095 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:30.713757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:53135)
0927 03:56:30.713946 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:30.714005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260927 03:56:31.527383 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:31.527038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:59975)
0927 03:56:31.527244 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:31.527313 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 03:56:31.580330 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:31.580019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:57799)
0927 03:56:31.580198 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:31.580260 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260927 03:56:31.715253 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:31.714868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:59423)
0927 03:56:31.715118 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:31.715182 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":126,"threads_started":1}
I20260927 03:56:32.528467 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:32.528145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:49015)
0927 03:56:32.528315 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:32.528355 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260927 03:56:32.581422 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:32.581042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:38591)
0927 03:56:32.581266 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:32.581346 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260927 03:56:32.716230 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:32.715895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:35247)
0927 03:56:32.716091 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:32.716161 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260927 03:56:33.529525 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:33.529213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:42965)
0927 03:56:33.529414 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:33.529475 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260927 03:56:33.582485 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:33.582164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:44779)
0927 03:56:33.582370 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:33.582426 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260927 03:56:33.717280 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:33.716970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:42193)
0927 03:56:33.717140 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:33.717214 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260927 03:56:34.281764 16516 raft_consensus.cc:482] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:34.281924 16516 raft_consensus.cc:504] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383dba6283134ef6a05cf94263123de4" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38633 } } peers { permanent_uuid: "39c2606f77f64a77be8c795307ea62bb" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40379 } } peers { permanent_uuid: "90b1952eee954485a75edef87a2844d9" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 39185 } }
I20260927 03:56:34.282291 16516 leader_election.cc:288] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633), 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379)
W20260927 03:56:34.282989 16097 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 383dba6283134ef6a05cf94263123de4 (127.1.50.66:38633): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
W20260927 03:56:34.283254 16095 leader_election.cc:341] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c2606f77f64a77be8c795307ea62bb (127.1.50.67:40379): Not found: Tablet not found: ee0a04bac0bc4f4d98cc3a61f40c1f2d
I20260927 03:56:34.283372 16095 leader_election.cc:302] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1952eee954485a75edef87a2844d9; no voters: 383dba6283134ef6a05cf94263123de4, 39c2606f77f64a77be8c795307ea62bb
I20260927 03:56:34.283463 16516 raft_consensus.cc:2726] T ee0a04bac0bc4f4d98cc3a61f40c1f2d P 90b1952eee954485a75edef87a2844d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:56:34.530722 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:34.530334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:56975)
0927 03:56:34.530552 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:34.530624 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260927 03:56:34.583508 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:34.583162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:32819)
0927 03:56:34.583356 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:34.583422 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260927 03:56:34.718274 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:34.717895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:38669)
0927 03:56:34.718141 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:34.718208 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":66,"threads_started":1}
I20260927 03:56:35.531747 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:35.531446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:51341)
0927 03:56:35.531622 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:35.531686 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260927 03:56:35.584378 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:35.584075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:58539)
0927 03:56:35.584248 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:35.584313 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260927 03:56:35.719173 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:35.718920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:51597)
0927 03:56:35.719081 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:35.719121 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260927 03:56:36.532721 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:36.532315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:34277)
0927 03:56:36.532580 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:36.532652 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":68,"threads_started":1}
I20260927 03:56:36.585374 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:36.585056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:59161)
0927 03:56:36.585235 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:36.585307 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260927 03:56:36.720153 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:36.719872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:49495)
0927 03:56:36.720052 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:36.720105 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260927 03:56:37.533741 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:37.533429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:59601)
0927 03:56:37.533629 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:37.533675 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260927 03:56:37.586305 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:37.586014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:56415)
0927 03:56:37.586199 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:37.586240 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260927 03:56:37.721159 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:37.720824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.67:44789)
0927 03:56:37.721047 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:37.721111 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260927 03:56:38.534749 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:38.534472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.65:36285)
0927 03:56:38.534634 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:38.534690 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260927 03:56:38.587276 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:38.586950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:34767 (local address 127.1.50.66:58225)
0927 03:56:38.587152 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:38.587214 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260927 03:56:38.709348 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16050
I20260927 03:56:38.715016 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16213
I20260927 03:56:38.719578 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16346
2026-09-27T03:56:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18210 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:56:38 dist-test-slave-96fg krb5kdc[16533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:56:38 dist-test-slave-96fg krb5kdc[16533](info): set up 2 sockets
Sep 27 03:56:38 dist-test-slave-96fg krb5kdc[16533](info): commencing operation
krb5kdc: starting...
W20260927 03:56:39.697897 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:56:39 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:56:39.821822 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:39.905834 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:39.906111 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:39.906139 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:39.907418 16546 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:56:39.907473 16546 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:56:39.907487 16546 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:56:39.907500 16546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:56:39.907513 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:39.907527 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:39.907554 16546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:56:39.907581 16546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:56:39.909247 16546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:39.909528 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:39.909790 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:39.915122 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:39.915153 16546 server_base.cc:1034] running on GCE node
W20260927 03:56:39.915182 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:39.915122 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:39.915681 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:39.915974 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:39.917193 16546 hybrid_clock.cc:648] HybridClock initialized: now 1790481399917167 us; error 38 us; skew 500 ppm
Sep 27 03:56:39 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481399, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:39.920212 16546 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:56:39.920648 16546 webserver.cc:460] Webserver started at http://127.1.50.126:38203/ using document root <none> and password file <none>
I20260927 03:56:39.920868 16546 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:39.921000 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:39.921175 16546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:39.922386 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "4b10dc4693714e3fb94d1ee8bd8a2008"
format_stamp: "Formatted at 2026-09-27 03:56:39 on dist-test-slave-96fg"
server_key: "e2ef70bc68aa0aed838285cd09185cb9"
server_key_iv: "2d71e24bbde301ff3ef86cd215f0f725"
server_key_version: "encryptionkey@0"
I20260927 03:56:39.922734 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "4b10dc4693714e3fb94d1ee8bd8a2008"
format_stamp: "Formatted at 2026-09-27 03:56:39 on dist-test-slave-96fg"
server_key: "e2ef70bc68aa0aed838285cd09185cb9"
server_key_iv: "2d71e24bbde301ff3ef86cd215f0f725"
server_key_version: "encryptionkey@0"
I20260927 03:56:39.924096 16546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:56:39.924937 16561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:39.925170 16546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:39.925282 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "4b10dc4693714e3fb94d1ee8bd8a2008"
format_stamp: "Formatted at 2026-09-27 03:56:39 on dist-test-slave-96fg"
server_key: "e2ef70bc68aa0aed838285cd09185cb9"
server_key_iv: "2d71e24bbde301ff3ef86cd215f0f725"
server_key_version: "encryptionkey@0"
I20260927 03:56:39.925354 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:39.936362 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:39.938656 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:39.938848 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:39.942999 16546 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:36067
I20260927 03:56:39.943046 16613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:36067 every 8 connection(s)
I20260927 03:56:39.943403 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:56:39.944278 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:39.946708 16614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap starting.
I20260927 03:56:39.947577 16614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:39.947777 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16546
I20260927 03:56:39.947889 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:56:39.947911 16614 log.cc:824] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:39.948098 1225 external_mini_cluster.cc:1428] Setting key c8c55a96428020c7a9a8afe723327693
I20260927 03:56:39.948772 16614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: No bootstrap required, opened a new log
I20260927 03:56:39.951232 16614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:56:39.951393 16614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:39.951428 16614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Initialized, Role: FOLLOWER
I20260927 03:56:39.951550 16614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:56:39.951628 16614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:56:39.951660 16614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:56:39.951706 16614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:39.952304 16614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:56:39.952402 16614 leader_election.cc:302] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b10dc4693714e3fb94d1ee8bd8a2008; no voters:
I20260927 03:56:39.952576 16614 leader_election.cc:288] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:56:39.952616 16619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:39.952757 16619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 LEADER]: Becoming Leader. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Running, Role: LEADER
I20260927 03:56:39.952906 16614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:56:39.952893 16619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:56:39.953331 16620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:56:39.953410 16620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:39.953629 16621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b10dc4693714e3fb94d1ee8bd8a2008. Latest consensus state: current_term: 1 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:56:39.953739 16621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:56:39.953895 16626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:56:39.954443 16626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:56:39 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:39.956080 16626 catalog_manager.cc:1347] Generated new cluster ID: 13d778b610cb4703811764fced951627
I20260927 03:56:39.956140 16626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:56:39.959254 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:39.948657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.1.50.126:36067)
0927 03:56:39.948851 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:39.948861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:39.948895 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:56:39.949363 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:39.949369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:39.949420 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:39.949543 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:39.950210 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:39.951126 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:39.952033 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:39.952493 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:39.952600 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:39.955412 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:39.955416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:39.955417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:39.955429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:39.957392 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:39.957851 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:39.957854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:39.957854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:39.957955 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:39.958197 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:39.958198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:39.958198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:39.958543 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:39.958709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:39.958828 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:56:39.958983 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260927 03:56:39.962826 16626 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:56:39.963342 16626 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:56:39.977142 16626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Generated new TSK 0
I20260927 03:56:39.977387 16626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:56:40.006424 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:40.090572 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:40.090847 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:40.090879 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:40.092137 16640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:40.092187 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:40.092216 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:56:40.093771 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:40.094008 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:40.094277 16640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:40.099481 16645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:40.099557 16640 server_base.cc:1034] running on GCE node
W20260927 03:56:40.099473 16648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:40.099473 16646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:40.099840 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:40.100126 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:40.101269 16640 hybrid_clock.cc:648] HybridClock initialized: now 1790481400101249 us; error 30 us; skew 500 ppm
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:40.104210 16640 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:56:40.104939 16640 webserver.cc:460] Webserver started at http://127.1.50.65:41167/ using document root <none> and password file <none>
I20260927 03:56:40.105175 16640 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:40.105226 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:40.105355 16640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:40.106500 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "0293997e9e75475da428172e476c0828"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "e1ffab299f5806f5d4a90831510197b3"
server_key_iv: "e957f4d9de536a653f4ad072946a56ee"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.106824 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "0293997e9e75475da428172e476c0828"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "e1ffab299f5806f5d4a90831510197b3"
server_key_iv: "e957f4d9de536a653f4ad072946a56ee"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.108101 16640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:56:40.108822 16655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.108959 16640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:56:40.109040 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "0293997e9e75475da428172e476c0828"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "e1ffab299f5806f5d4a90831510197b3"
server_key_iv: "e957f4d9de536a653f4ad072946a56ee"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.109102 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:40.144546 16640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:40.146711 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:40.146857 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:40.147090 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:40.147394 16640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:40.147428 16640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.147457 16640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:40.147475 16640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.153146 16640 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:41605
I20260927 03:56:40.153256 16768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:41605 every 8 connection(s)
I20260927 03:56:40.153559 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:40.162164 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16640
I20260927 03:56:40.162322 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:56:40.162539 1225 external_mini_cluster.cc:1428] Setting key cbd58103b5722cdffe83221b7b2bbd99
I20260927 03:56:40.163559 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.154332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55085 (local address 127.1.50.126:36067)
0927 03:56:40.154391 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.154394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.154612 (+ 218us) server_negotiation.cc:407] Connection header received
0927 03:56:40.155157 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.155158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.155190 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.155239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:40.156155 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.156808 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.157872 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.158105 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.158172 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.160443 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:40.160444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:40.160444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:40.160452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:40.161971 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.162514 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.162515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.162516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.162581 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.162853 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.162854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.162854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.162978 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:40.163038 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.163358 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.163468 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:56:40.163721 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.154315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:55085)
0927 03:56:40.154494 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.154532 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.155063 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.155301 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.155304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.155866 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:40.156060 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.156064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.156907 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.156908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.157771 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.157774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.158187 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.158188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.158293 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.158841 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:40.158852 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:40.160345 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:40.162101 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.162111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.162114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.162430 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.162667 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.162668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.162668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.162786 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.163055 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:40.163063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:40.163220 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.163323 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.163492 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260927 03:56:40.164203 16769 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:56:40.164314 16769 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:40.164521 16769 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:56:40.165108 16578 ts_manager.cc:194] Registered new tserver with Master: 0293997e9e75475da428172e476c0828 (127.1.50.65:41605)
I20260927 03:56:40.165711 16578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:55085
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:56:40.189651 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:40.273002 16774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:40.273239 16774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:40.273268 16774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:40.274576 16774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:40.274623 16774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:40.274655 16774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:56:40.276154 16774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:40.276364 16774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:40.276561 16774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:56:40.282011 16780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:40.282091 16774 server_base.cc:1034] running on GCE node
W20260927 03:56:40.282061 16779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:40.282059 16782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:40.282531 16774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:40.282790 16774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:40.283962 16774 hybrid_clock.cc:648] HybridClock initialized: now 1790481400283940 us; error 35 us; skew 500 ppm
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:40.286669 16774 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:56:40.287353 16774 webserver.cc:460] Webserver started at http://127.1.50.66:37299/ using document root <none> and password file <none>
I20260927 03:56:40.287590 16774 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:40.287642 16774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:40.287781 16774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:40.288872 16774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "db9fe502c81b462ea85f8ad72c7a950a"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "854d1b1284bf122588643c0876862159"
server_key_iv: "78206891b26346ebf6d101c496a2730b"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.289225 16774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "db9fe502c81b462ea85f8ad72c7a950a"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "854d1b1284bf122588643c0876862159"
server_key_iv: "78206891b26346ebf6d101c496a2730b"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.290809 16774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:56:40.291908 16789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.292227 16774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:56:40.292351 16774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "db9fe502c81b462ea85f8ad72c7a950a"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "854d1b1284bf122588643c0876862159"
server_key_iv: "78206891b26346ebf6d101c496a2730b"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.292430 16774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:40.311329 16774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:40.313894 16774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:40.314101 16774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:40.314368 16774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:40.314679 16774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:40.314713 16774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.314744 16774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:40.314769 16774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.320901 16774 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:35831
I20260927 03:56:40.320983 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:35831 every 8 connection(s)
I20260927 03:56:40.321288 16774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:56:40.325095 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16774
I20260927 03:56:40.325208 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:56:40.325489 1225 external_mini_cluster.cc:1428] Setting key af673138ae95380fa24e16225cac0b73
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:40.331861 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.322151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:33039 (local address 127.1.50.126:36067)
0927 03:56:40.322209 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.322212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.322454 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:56:40.323027 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.323028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.323039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.323096 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:40.324085 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.324633 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.326191 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.326443 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.326507 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.328780 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:40.328782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:40.328783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:40.328791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:40.330463 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.330899 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.330900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.330900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.330962 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.331190 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.331191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.331191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.331322 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:40.331379 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.331657 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.331765 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:56:40.332037 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.322142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:33039)
0927 03:56:40.322310 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.322357 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.322893 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.323151 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.323155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.323730 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:40.323947 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.323952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.324751 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.324752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.326096 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.326101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.326514 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.326515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.326672 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.327237 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:40.327246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:40.328670 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:40.330559 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.330571 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.330574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.330824 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.331020 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.331021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.331021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.331130 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.331380 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:40.331387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:40.331535 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.331629 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.331809 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260927 03:56:40.332839 16903 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:56:40.332933 16903 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:40.333122 16903 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:56:40.333616 16578 ts_manager.cc:194] Registered new tserver with Master: db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:56:40.334131 16578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:33039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:56:40.358343 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:56:40.443948 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:56:40.444172 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:56:40.444198 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:56:40.445453 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:56:40.445497 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:56:40.445525 16908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:56:40.447119 16908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:56:40.447363 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:56:40.447892 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:56:40.453639 16908 server_base.cc:1034] running on GCE node
W20260927 03:56:40.453749 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:40.453809 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:56:40.453678 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:56:40.454171 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:56:40.454440 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:56:40.455590 16908 hybrid_clock.cc:648] HybridClock initialized: now 1790481400455559 us; error 47 us; skew 500 ppm
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:56:40.458194 16908 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:56:40.458792 16908 webserver.cc:460] Webserver started at http://127.1.50.67:43141/ using document root <none> and password file <none>
I20260927 03:56:40.459026 16908 fs_manager.cc:359] Metadata directory not provided
I20260927 03:56:40.459080 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:56:40.459225 16908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:56:40.460290 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "99ad96e5bb05411c9ce9b32b17c907fa"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "97e9fad4d42b8f3a1e9458f2511b44b4"
server_key_iv: "f4aa17a911c30ad6d89e649cb7f76e45"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.460655 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "99ad96e5bb05411c9ce9b32b17c907fa"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "97e9fad4d42b8f3a1e9458f2511b44b4"
server_key_iv: "f4aa17a911c30ad6d89e649cb7f76e45"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.462091 16908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:56:40.462895 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.463068 16908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:56:40.463156 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "99ad96e5bb05411c9ce9b32b17c907fa"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "97e9fad4d42b8f3a1e9458f2511b44b4"
server_key_iv: "f4aa17a911c30ad6d89e649cb7f76e45"
server_key_version: "encryptionkey@0"
I20260927 03:56:40.463230 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:56:40.485029 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:56:40.487145 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:56:40.487267 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:56:40.487475 16908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:56:40.487739 16908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:56:40.487767 16908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.487788 16908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:56:40.487804 16908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:56:40.493535 16908 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40013
I20260927 03:56:40.493571 17036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40013 every 8 connection(s)
I20260927 03:56:40.493913 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:56:40 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:56:40.503182 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.494655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:59275 (local address 127.1.50.126:36067)
0927 03:56:40.494708 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.494710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.494939 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:56:40.495443 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.495445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.495458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.495509 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:40.496543 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.497078 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.498062 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.498259 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.498313 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.500394 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:40.500395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:40.500395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:40.500404 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:40.501783 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.502261 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.502262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.502262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.502318 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.502598 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.502598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.502599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.502719 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:40.502772 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.503042 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.503110 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:56:40.503391 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.494621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:59275)
0927 03:56:40.494808 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.494856 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.495350 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.495573 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.495578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.496201 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:56:40.496450 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.496456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.497163 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.497164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.497950 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.497952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.498336 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.498337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.498445 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.498925 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:56:40.498933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:56:40.500268 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:56:40.501894 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.501901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.501903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.502183 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.502421 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:56:40.502422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:56:40.502422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:56:40.502541 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:56:40.502789 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:56:40.502794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:56:40.502928 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.503016 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.503147 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260927 03:56:40.503769 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 16908
I20260927 03:56:40.503859 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:56:40.504014 17037 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:56:40.504061 1225 external_mini_cluster.cc:1428] Setting key bdc3d0fefe01a51034be72d87b316e9e
I20260927 03:56:40.504120 17037 heartbeater.cc:464] Registering TS with master...
I20260927 03:56:40.504323 17037 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:56:40.504745 16577 ts_manager.cc:194] Registered new tserver with Master: 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013)
I20260927 03:56:40.505167 16577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:59275
I20260927 03:56:40.516063 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:56:40.524430 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.517528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.1.50.126:36067)
0927 03:56:40.517580 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.517582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.517678 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:56:40.517829 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.517830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.517840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.517897 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:56:40.518524 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.519011 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.519771 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.519982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.520052 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.521252 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:56:40.521253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:56:40.521253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:56:40.521260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:56:40.522865 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.523439 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.523441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.523441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.523525 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:56:40.523716 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:56:40.523717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:56:40.523718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:56:40.523889 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:56:40.523982 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.524171 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.524356 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:56:40.526477 16577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:56:40.528152 16577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:56:40.535204 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.530634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40013 (local address 127.0.0.1:47254)
0927 03:56:40.530747 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.530789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.530851 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.531610 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.531612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.531613 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:40.531640 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.531641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532977 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.532978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.533958 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.533961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.534948 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.535035 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.535038 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.535084 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.535116 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":24,"thread_start_us":56,"threads_started":1}
I20260927 03:56:40.535504 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.531068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.1.50.66:35831)
0927 03:56:40.531266 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.531272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.531283 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:40.531325 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.531327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.531344 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.531434 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:40.531608 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532473 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.533948 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.534578 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534669 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.534740 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.535363 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.535419 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260927 03:56:40.535496 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.530644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.0.0.1:45998)
0927 03:56:40.530807 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.530818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.530871 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.531466 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.531467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.531468 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:40.531548 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.531550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532952 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.532953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.534107 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.534109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.535018 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.535019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.535114 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.535118 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.535166 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.535215 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":34,"thread_start_us":99,"threads_started":1}
I20260927 03:56:40.535802 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.531088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.1.50.67:40013)
0927 03:56:40.531266 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.531272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.531283 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:56:40.531325 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.531327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.531344 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.531434 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:40.531710 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532835 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534096 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.534822 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534900 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.534961 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.535598 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.535653 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260927 03:56:40.535830 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.530928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.1.50.65:41605)
0927 03:56:40.531276 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.531279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.531289 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:56:40.531322 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.531323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.531336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.531626 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:40.531702 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532852 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534218 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.534953 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.535015 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.535071 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.535192 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.535237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":94,"threads_started":1}
I20260927 03:56:40.535205 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.530823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.0.0.1:50624)
0927 03:56:40.530989 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.530999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.531038 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.531456 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.531459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.531462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:40.531548 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.531550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.532579 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.532580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.533831 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.533834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.534670 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.534671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.534899 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.534936 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.535023 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.535093 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":129,"threads_started":1}
I20260927 03:56:40.536550 16971 tablet_service.cc:1467] Processing CreateTablet for tablet c936d4ce2e6842e682376c3d882d137f (DEFAULT_TABLE table=test-table [id=e6dbb91c41064394937eb5f5be0b502e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:40.536550 16703 tablet_service.cc:1467] Processing CreateTablet for tablet c936d4ce2e6842e682376c3d882d137f (DEFAULT_TABLE table=test-table [id=e6dbb91c41064394937eb5f5be0b502e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:40.536796 16837 tablet_service.cc:1467] Processing CreateTablet for tablet c936d4ce2e6842e682376c3d882d137f (DEFAULT_TABLE table=test-table [id=e6dbb91c41064394937eb5f5be0b502e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:56:40.536906 16971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c936d4ce2e6842e682376c3d882d137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:40.536904 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c936d4ce2e6842e682376c3d882d137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:40.537040 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c936d4ce2e6842e682376c3d882d137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:56:40.539839 17054 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Bootstrap starting.
I20260927 03:56:40.540189 17055 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Bootstrap starting.
I20260927 03:56:40.540531 17054 tablet_bootstrap.cc:654] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:40.540808 17055 tablet_bootstrap.cc:654] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:40.541131 17054 log.cc:824] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:40.541342 17055 log.cc:824] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:40.541671 17056 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Bootstrap starting.
I20260927 03:56:40.541993 17054 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: No bootstrap required, opened a new log
I20260927 03:56:40.542068 17055 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: No bootstrap required, opened a new log
I20260927 03:56:40.542094 17054 ts_tablet_manager.cc:1397] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:56:40.542129 17055 ts_tablet_manager.cc:1397] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:56:40.542352 17056 tablet_bootstrap.cc:654] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Neither blocks nor log segments found. Creating new log.
I20260927 03:56:40.542903 17056 log.cc:824] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Log is configured to *not* fsync() on all Append() calls
I20260927 03:56:40.543692 17056 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: No bootstrap required, opened a new log
I20260927 03:56:40.543777 17056 ts_tablet_manager.cc:1397] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:56:40.544003 17054 raft_consensus.cc:348] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.544138 17054 raft_consensus.cc:374] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:40.544158 17054 raft_consensus.cc:729] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9fe502c81b462ea85f8ad72c7a950a, State: Initialized, Role: FOLLOWER
I20260927 03:56:40.544257 17054 consensus_queue.cc:260] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.544143 17055 raft_consensus.cc:348] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.544346 17055 raft_consensus.cc:374] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:40.544384 17055 raft_consensus.cc:729] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ad96e5bb05411c9ce9b32b17c907fa, State: Initialized, Role: FOLLOWER
I20260927 03:56:40.544528 17054 ts_tablet_manager.cc:1428] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:56:40.544518 17055 consensus_queue.cc:260] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.544533 16903 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
I20260927 03:56:40.544796 17055 ts_tablet_manager.cc:1428] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:56:40.544817 17037 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
I20260927 03:56:40.545845 17056 raft_consensus.cc:348] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.545989 17056 raft_consensus.cc:374] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:56:40.546008 17056 raft_consensus.cc:729] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0293997e9e75475da428172e476c0828, State: Initialized, Role: FOLLOWER
I20260927 03:56:40.546149 17056 consensus_queue.cc:260] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.546401 17056 ts_tablet_manager.cc:1428] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:56:40.546464 16769 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
W20260927 03:56:40.571915 16904 tablet.cc:2369] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:56:40.654312 16770 tablet.cc:2369] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:56:40.744509 17038 tablet.cc:2369] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:56:40.763453 17061 raft_consensus.cc:482] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:56:40.763576 17061 raft_consensus.cc:504] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.763900 17061 leader_election.cc:288] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0293997e9e75475da428172e476c0828 (127.1.50.65:41605), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
I20260927 03:56:40.767642 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.764279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:56333 (local address 127.1.50.66:35831)
0927 03:56:40.764342 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.764344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.764543 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:56:40.764591 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.764592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.764605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.764653 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:40.764846 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.765325 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.766638 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.767213 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.767296 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.767309 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.767519 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.767562 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:56:40.767781 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.764206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.1.50.67:56333)
0927 03:56:40.764471 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.764481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.764535 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.764668 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.764670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.764672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:40.764781 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.764783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.765411 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.765412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.766521 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.766525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.767327 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.767329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.767423 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.767431 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.767507 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.767572 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":36,"thread_start_us":138,"threads_started":1}
I20260927 03:56:40.767854 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.764339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:58537 (local address 127.1.50.65:41605)
0927 03:56:40.764424 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.764427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.764442 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:56:40.764478 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.764479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.764493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.764566 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:56:40.764768 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.765309 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.766698 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.767273 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.767339 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.767359 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.767732 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.767782 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:56:40.768004 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.764181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.1.50.67:58537)
0927 03:56:40.764261 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.764278 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 03:56:40.764359 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:56:40.764595 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:56:40.764597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:56:40.764600 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:56:40.764697 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.764699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.765390 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.765391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.766591 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:56:40.766595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.767401 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.767402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:56:40.767508 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.767513 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:56:40.767562 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:56:40.767622 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":41}
I20260927 03:56:40.768404 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c936d4ce2e6842e682376c3d882d137f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9fe502c81b462ea85f8ad72c7a950a" is_pre_election: true
I20260927 03:56:40.768472 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c936d4ce2e6842e682376c3d882d137f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0293997e9e75475da428172e476c0828" is_pre_election: true
I20260927 03:56:40.768626 16857 raft_consensus.cc:2454] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 0.
I20260927 03:56:40.768723 16723 raft_consensus.cc:2454] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 0.
I20260927 03:56:40.768949 16925 leader_election.cc:302] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a; no voters:
I20260927 03:56:40.769081 17061 raft_consensus.cc:2781] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:56:40.769174 17061 raft_consensus.cc:482] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:56:40.769230 17061 raft_consensus.cc:3037] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:40.770263 17061 raft_consensus.cc:504] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.770431 17061 leader_election.cc:288] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 election: Requested vote from peers 0293997e9e75475da428172e476c0828 (127.1.50.65:41605), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
I20260927 03:56:40.770624 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c936d4ce2e6842e682376c3d882d137f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9fe502c81b462ea85f8ad72c7a950a"
I20260927 03:56:40.770727 16857 raft_consensus.cc:3037] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:40.770913 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c936d4ce2e6842e682376c3d882d137f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0293997e9e75475da428172e476c0828"
I20260927 03:56:40.771118 16723 raft_consensus.cc:3037] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:56:40.771569 16857 raft_consensus.cc:2454] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 1.
I20260927 03:56:40.771777 16925 leader_election.cc:302] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a; no voters:
I20260927 03:56:40.771874 17061 raft_consensus.cc:2781] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:56:40.772047 17061 raft_consensus.cc:686] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 LEADER]: Becoming Leader. State: Replica: 99ad96e5bb05411c9ce9b32b17c907fa, State: Running, Role: LEADER
I20260927 03:56:40.772156 17061 consensus_queue.cc:237] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:56:40.772495 16723 raft_consensus.cc:2454] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 1.
I20260927 03:56:40.773171 16577 catalog_manager.cc:5495] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67). New cstate: current_term: 1 leader_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260927 03:56:40.821733 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.818983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.1.50.67:40013)
0927 03:56:40.819115 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 03:56:40.819118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:56:40.819128 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:56:40.819176 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:56:40.819177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:56:40.819204 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:56:40.819280 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:56:40.819417 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.819975 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.820738 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:56:40.820945 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:56:40.821010 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:56:40.821237 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:56:40.821396 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:56:40.821554 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:56:40.821572 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:56:40.821602 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 03:56:40.824039 16857 raft_consensus.cc:1261] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Refusing update from remote peer 99ad96e5bb05411c9ce9b32b17c907fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:40.824082 16723 raft_consensus.cc:1261] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Refusing update from remote peer 99ad96e5bb05411c9ce9b32b17c907fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:56:40.824471 17064 consensus_queue.cc:1035] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:40.824623 17061 consensus_queue.cc:1035] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:56:40.827534 17067 mvcc.cc:205] Tried to move back new op lower bound from 7333811817772306432 to 7333811817563381760. Current Snapshot: MvccSnapshot[applied={T|T < 7333811817772306432}]
I20260927 03:56:40.828289 17069 mvcc.cc:205] Tried to move back new op lower bound from 7333811817772306432 to 7333811817563381760. Current Snapshot: MvccSnapshot[applied={T|T < 7333811817772306432}]
I20260927 03:56:40.830955 16577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:40.831115 16577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:56:40.832089 16577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Sending DeleteTablet for 3 replicas of tablet c936d4ce2e6842e682376c3d882d137f
I20260927 03:56:40.832456 16703 tablet_service.cc:1514] Processing DeleteTablet for tablet c936d4ce2e6842e682376c3d882d137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:40 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:45998
I20260927 03:56:40.832548 16971 tablet_service.cc:1514] Processing DeleteTablet for tablet c936d4ce2e6842e682376c3d882d137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:40 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:47254
I20260927 03:56:40.832635 16837 tablet_service.cc:1514] Processing DeleteTablet for tablet c936d4ce2e6842e682376c3d882d137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:40 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:50624
I20260927 03:56:40.832852 17078 tablet_replica.cc:330] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: stopping tablet replica
I20260927 03:56:40.832872 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16640
I20260927 03:56:40.832991 17078 raft_consensus.cc:2229] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:56:40.833114 17078 raft_consensus.cc:2258] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:40.838224 17079 tablet_replica.cc:330] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: stopping tablet replica
I20260927 03:56:40.838587 17077 tablet_replica.cc:330] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: stopping tablet replica
I20260927 03:56:40.838423 17079 raft_consensus.cc:2229] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 03:56:40.838758 16564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:41605 (error 108)
I20260927 03:56:40.838810 17079 raft_consensus.cc:2258] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:40.838917 17077 raft_consensus.cc:2229] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:56:40.839380 17077 raft_consensus.cc:2258] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:56:40.839440 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16774
I20260927 03:56:40.839473 17079 ts_tablet_manager.cc:1905] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:40.842010 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.841834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.0.0.1:60658)
0927 03:56:40.841901 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.841966 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 03:56:40.842299 16564 catalog_manager.cc:4584] TS 0293997e9e75475da428172e476c0828 (127.1.50.65:41605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c936d4ce2e6842e682376c3d882d137f: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41605: connect: Connection refused (error 111)
I20260927 03:56:40.842486 17077 ts_tablet_manager.cc:1905] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:56:40.845018 17077 ts_tablet_manager.cc:1918] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:56:40.845108 17077 log.cc:1196] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/c936d4ce2e6842e682376c3d882d137f
I20260927 03:56:40.845381 17077 ts_tablet_manager.cc:1939] T c936d4ce2e6842e682376c3d882d137f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting consensus metadata
I20260927 03:56:40.845677 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:56:40.845529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.0.0.1:53212)
0927 03:56:40.845598 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:56:40.845640 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.66:35831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:56:40.845836 16563 catalog_manager.cc:4841] TS 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): tablet c936d4ce2e6842e682376c3d882d137f (table test-table [id=e6dbb91c41064394937eb5f5be0b502e]) successfully deleted
I20260927 03:56:40.846158 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16908
I20260927 03:56:40.851475 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 16546
I20260927 03:57:00.858336 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--webserver_interface=127.1.50.126
--webserver_port=38203
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:00.941783 17081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:00.942009 17081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:00.942060 17081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:00.943392 17081 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:57:00.943462 17081 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:57:00.943475 17081 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:57:00.943486 17081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:00.943498 17081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:00.943511 17081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:00.943523 17081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:00.943547 17081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:00.945361 17081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=38203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:00.945634 17081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:00.945887 17081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:57:00.955588 17081 server_base.cc:1034] running on GCE node
W20260927 03:57:00.955509 17087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:00.955607 17090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:00.955509 17088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:00.956234 17081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:00.956663 17081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:00.957881 17081 hybrid_clock.cc:648] HybridClock initialized: now 1790481420957854 us; error 46 us; skew 500 ppm
Sep 27 03:57:00 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481420, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:00.964934 17081 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:57:00.965523 17081 webserver.cc:460] Webserver started at http://127.1.50.126:38203/ using document root <none> and password file <none>
I20260927 03:57:00.965822 17081 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:00.965910 17081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:00.967680 17081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:00.968595 17097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:00.968884 17081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:00.968989 17081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "4b10dc4693714e3fb94d1ee8bd8a2008"
format_stamp: "Formatted at 2026-09-27 03:56:39 on dist-test-slave-96fg"
server_key: "e2ef70bc68aa0aed838285cd09185cb9"
server_key_iv: "2d71e24bbde301ff3ef86cd215f0f725"
server_key_version: "encryptionkey@0"
I20260927 03:57:00.969502 17081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:00.998819 17081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:01.001230 17081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:01.001415 17081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:01.006384 17081 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:36067
I20260927 03:57:01.006445 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:36067 every 8 connection(s)
I20260927 03:57:01.006889 17081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:01.008563 17150 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:57:01.009318 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap starting.
I20260927 03:57:01.015462 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17081
I20260927 03:57:01.015837 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:41605
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=41167
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:57:01.018282 17150 log.cc:824] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:01.021066 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:57:01.021431 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap complete.
I20260927 03:57:01.024776 17150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:01.025121 17150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.025305 17150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:01.025447 17150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:01.025532 17150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:01.025590 17150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:57:01.027115 17150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:01.027243 17150 leader_election.cc:302] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b10dc4693714e3fb94d1ee8bd8a2008; no voters:
I20260927 03:57:01.027443 17150 leader_election.cc:288] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:57:01.027580 17154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:57:01.027732 17150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:01.027889 17154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 LEADER]: Becoming Leader. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Running, Role: LEADER
I20260927 03:57:01.028097 17154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:01.028966 17156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b10dc4693714e3fb94d1ee8bd8a2008. Latest consensus state: current_term: 2 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:57:01.028923 17155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:57:01.029080 17156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:01.029089 17155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:01.030277 17169 catalog_manager.cc:1259] Loaded cluster ID: 13d778b610cb4703811764fced951627
I20260927 03:57:01.030342 17169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: loading cluster ID for follower catalog manager: success
I20260927 03:57:01.031636 17169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: acquiring CA information for follower catalog manager: success
I20260927 03:57:01.032081 17169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:57:01.032181 17167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:01.032707 17167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:01.032789 17167 catalog_manager.cc:1259] Loaded cluster ID: 13d778b610cb4703811764fced951627
I20260927 03:57:01.032820 17167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:01.033000 17167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:01.033082 17167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Loaded TSK: 0
I20260927 03:57:01.038478 17167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Generated new TSK 1
I20260927 03:57:01.038576 17167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:57:01.125216 17152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:01.125459 17152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:01.125490 17152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:01.127107 17152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:01.127168 17152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:01.127208 17152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:01.129050 17152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:41605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=41167
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:01.129326 17152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:01.129565 17152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:01.134842 17178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.134853 17177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.134855 17180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:01.135474 17152 server_base.cc:1034] running on GCE node
I20260927 03:57:01.135725 17152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:01.136021 17152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:01.137197 17152 hybrid_clock.cc:648] HybridClock initialized: now 1790481421137185 us; error 30 us; skew 500 ppm
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:01.140398 17152 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:57:01.140972 17152 webserver.cc:460] Webserver started at http://127.1.50.65:41167/ using document root <none> and password file <none>
I20260927 03:57:01.141247 17152 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:01.141320 17152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:01.143173 17152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.143982 17187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.144171 17152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.144266 17152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "0293997e9e75475da428172e476c0828"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "e1ffab299f5806f5d4a90831510197b3"
server_key_iv: "e957f4d9de536a653f4ad072946a56ee"
server_key_version: "encryptionkey@0"
I20260927 03:57:01.144774 17152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:01.164736 17152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:01.167577 17152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:01.167752 17152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:01.168030 17152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:01.168574 17194 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:57:01.170243 17152 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:57:01.170302 17152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:57:01.170336 17152 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:57:01.171164 17152 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:57:01.171205 17152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:01.171314 17194 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Bootstrap starting.
I20260927 03:57:01.178134 17152 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:41605
I20260927 03:57:01.178627 17152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:01.181632 17194 log.cc:824] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:01.182268 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:41605 every 8 connection(s)
I20260927 03:57:01.183096 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17152
I20260927 03:57:01.183436 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:35831
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=37299
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:57:01.184409 17194 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:57:01.184682 17194 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Bootstrap complete.
I20260927 03:57:01.185789 17194 ts_tablet_manager.cc:1397] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260927 03:57:01.187335 17194 raft_consensus.cc:348] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:57:01.187682 17194 raft_consensus.cc:729] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0293997e9e75475da428172e476c0828, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.187855 17194 consensus_queue.cc:260] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:57:01.188143 17194 ts_tablet_manager.cc:1428] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:01.194934 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.179531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:58011)
0927 03:57:01.182517 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.182551 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.183153 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.183830 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.183835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.184192 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:01.184465 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.184472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.186152 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.186153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.187410 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.187413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.188430 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.188431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.188554 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.189128 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:01.189138 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:01.190919 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:01.193153 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.193160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.193162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.193406 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.193735 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.193736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.193736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.193848 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.194389 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:01.194395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:01.194522 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.194618 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.194766 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2943,"thread_start_us":94,"threads_started":1}
I20260927 03:57:01.195050 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.179629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:58011 (local address 127.1.50.126:36067)
0927 03:57:01.179858 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.179872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.182680 (+ 2808us) server_negotiation.cc:407] Connection header received
0927 03:57:01.183437 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.183443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.183500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.183621 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.184610 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.185977 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.187487 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.188236 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.188337 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.191043 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.191048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.191049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.191063 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.193051 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.193492 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.193496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.193497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.193624 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.193919 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.193919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.193920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.194300 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.194393 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.194696 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.194852 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260927 03:57:01.195689 17302 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:57:01.195801 17302 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:01.196002 17302 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:57:01.196826 17114 ts_manager.cc:194] Registered new tserver with Master: 0293997e9e75475da428172e476c0828 (127.1.50.65:41605)
I20260927 03:57:01.197831 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:58011
I20260927 03:57:01.198841 17302 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481420, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.65@KRBTEST.COM
I20260927 03:57:01.206054 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.197405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.0.0.1:40492)
0927 03:57:01.197588 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.197622 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.197688 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.197985 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.197989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.198332 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:01.198388 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.198390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.199333 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.199334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.200280 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.200282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.200802 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.200803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.200884 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.201333 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:01.201338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:01.202920 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:01.204910 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.204918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.204920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.205150 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.205364 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.205365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.205366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.205423 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.205722 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:01.205727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:01.205777 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.205845 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.205948 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260927 03:57:01.206269 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.197664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.1.50.65:41605)
0927 03:57:01.197834 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.197838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.197847 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:01.197879 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.197881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.197894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.198452 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.198491 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.199232 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.200376 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.200718 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.200791 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.203010 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.203020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.203021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.203029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.204803 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.205235 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.205238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.205239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.205305 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.205475 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.205476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.205476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.205661 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.205738 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.205889 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.205990 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260927 03:57:01.206673 17225 tablet_service.cc:1514] Processing DeleteTablet for tablet c936d4ce2e6842e682376c3d882d137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:40 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:40492
I20260927 03:57:01.207384 17311 tablet_replica.cc:330] stopping tablet replica
I20260927 03:57:01.207492 17311 raft_consensus.cc:2229] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:01.207567 17311 raft_consensus.cc:2258] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:01.207881 17311 ts_tablet_manager.cc:1905] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:01.210111 17311 ts_tablet_manager.cc:1918] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:01.210180 17311 log.cc:1196] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal/wals/c936d4ce2e6842e682376c3d882d137f
I20260927 03:57:01.210368 17311 ts_tablet_manager.cc:1939] T c936d4ce2e6842e682376c3d882d137f P 0293997e9e75475da428172e476c0828: Deleting consensus metadata
I20260927 03:57:01.211246 17100 catalog_manager.cc:4841] TS 0293997e9e75475da428172e476c0828 (127.1.50.65:41605): tablet c936d4ce2e6842e682376c3d882d137f (table test-table [id=e6dbb91c41064394937eb5f5be0b502e]) successfully deleted
W20260927 03:57:01.277297 17307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:01.277535 17307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:01.277562 17307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:01.278884 17307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:01.278931 17307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:01.278960 17307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:01.280529 17307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:35831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=37299
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:01.280756 17307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:01.280985 17307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:01.286120 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.286237 17317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.286116 17319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:01.286317 17307 server_base.cc:1034] running on GCE node
I20260927 03:57:01.286566 17307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:01.286805 17307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:01.287951 17307 hybrid_clock.cc:648] HybridClock initialized: now 1790481421287940 us; error 31 us; skew 500 ppm
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:01.290558 17307 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:57:01.291054 17307 webserver.cc:460] Webserver started at http://127.1.50.66:37299/ using document root <none> and password file <none>
I20260927 03:57:01.291307 17307 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:01.291366 17307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:01.292675 17307 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.293346 17326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.293497 17307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:01.293566 17307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "db9fe502c81b462ea85f8ad72c7a950a"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "854d1b1284bf122588643c0876862159"
server_key_iv: "78206891b26346ebf6d101c496a2730b"
server_key_version: "encryptionkey@0"
I20260927 03:57:01.293939 17307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:01.323169 17307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:01.325129 17307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:01.325253 17307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:01.325480 17307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:01.325886 17333 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:57:01.327319 17307 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:57:01.327378 17307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:57:01.327414 17307 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:57:01.328035 17307 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:57:01.328068 17307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:01.328152 17333 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Bootstrap starting.
I20260927 03:57:01.329942 17333 log.cc:824] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:01.332254 17333 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:57:01.332513 17333 tablet_bootstrap.cc:492] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Bootstrap complete.
I20260927 03:57:01.332645 17333 ts_tablet_manager.cc:1397] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 03:57:01.333874 17333 raft_consensus.cc:348] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:57:01.334165 17333 raft_consensus.cc:729] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9fe502c81b462ea85f8ad72c7a950a, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.334381 17307 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:35831
I20260927 03:57:01.334369 17333 consensus_queue.cc:260] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } } peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } }
I20260927 03:57:01.334416 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:35831 every 8 connection(s)
I20260927 03:57:01.334654 17333 ts_tablet_manager.cc:1428] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260927 03:57:01.334769 17307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:01.341187 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17307
I20260927 03:57:01.341485 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:40013
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=43141
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:57:01.344417 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.335546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:44007 (local address 127.1.50.126:36067)
0927 03:57:01.335594 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.335596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.335774 (+ 178us) server_negotiation.cc:407] Connection header received
0927 03:57:01.335881 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.335883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.335897 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.335951 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.336562 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.337115 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.338271 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.338483 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.338546 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.341336 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.341338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.341338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.341346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.342880 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.343367 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.343369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.343369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.343436 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.343748 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.343749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.343749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.343895 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.343960 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.344228 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.344324 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:57:01.344530 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.335411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:44007)
0927 03:57:01.335629 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.335664 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.335819 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.335998 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.336001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.336274 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:01.336474 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.336479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.337249 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.337250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.338183 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.338194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.338577 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.338578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.338686 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.339622 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:01.339631 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:01.341211 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:01.342995 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.343008 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.343011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.343293 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.343549 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.343550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.343550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.343671 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.343963 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:01.343969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:01.344100 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.344190 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.344336 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260927 03:57:01.345026 17443 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:57:01.345119 17443 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:01.345283 17443 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:57:01.345860 17114 ts_manager.cc:194] Registered new tserver with Master: db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
I20260927 03:57:01.346473 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:44007
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481420, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.66@KRBTEST.COM
I20260927 03:57:01.354583 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.346337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.1.50.66:35831)
0927 03:57:01.346502 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.346507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.346516 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:01.346549 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.346552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.346565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.346629 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.346923 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.348005 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.348920 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.349421 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.349506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.351692 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.351702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.351702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.351710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.353319 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.353737 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.353741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.353742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.353808 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.353987 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.353988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.353988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.354179 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.354290 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.354412 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.354497 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260927 03:57:01.354604 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.346171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.0.0.1:43996)
0927 03:57:01.346232 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.346242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.346280 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.346631 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.346632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.346791 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:01.346845 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.346847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.348134 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.348135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.348805 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.348808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.349529 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.349531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.349601 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.350149 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:01.350155 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:01.351589 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:01.353407 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.353408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.353409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.353630 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.353874 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.353874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.353875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.353935 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.354264 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:01.354264 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:01.354315 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.354387 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.354499 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:57:01.355023 17376 tablet_service.cc:1514] Processing DeleteTablet for tablet c936d4ce2e6842e682376c3d882d137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:56:40 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:43996
I20260927 03:57:01.355700 17448 tablet_replica.cc:330] stopping tablet replica
I20260927 03:57:01.355814 17448 raft_consensus.cc:2229] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:01.355890 17448 raft_consensus.cc:2258] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:01.356186 17448 ts_tablet_manager.cc:1905] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:01.358487 17448 ts_tablet_manager.cc:1918] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:01.358563 17448 log.cc:1196] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/c936d4ce2e6842e682376c3d882d137f
I20260927 03:57:01.358747 17448 ts_tablet_manager.cc:1939] T c936d4ce2e6842e682376c3d882d137f P db9fe502c81b462ea85f8ad72c7a950a: Deleting consensus metadata
I20260927 03:57:01.359236 17099 catalog_manager.cc:4841] TS db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): tablet c936d4ce2e6842e682376c3d882d137f (table test-table [id=e6dbb91c41064394937eb5f5be0b502e]) successfully deleted
W20260927 03:57:01.428498 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:01.428755 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:01.428787 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:01.430137 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:01.430192 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:01.430222 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:01.431754 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:40013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=43141
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:01.431991 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:01.432227 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:01.437332 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.437395 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:01.437362 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:01.437427 17446 server_base.cc:1034] running on GCE node
I20260927 03:57:01.437739 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:01.437963 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:01.439110 17446 hybrid_clock.cc:648] HybridClock initialized: now 1790481421439095 us; error 30 us; skew 500 ppm
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:01.446563 17446 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:57:01.447130 17446 webserver.cc:460] Webserver started at http://127.1.50.67:43141/ using document root <none> and password file <none>
I20260927 03:57:01.447350 17446 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:01.447414 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:01.448832 17446 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.449550 17463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.449714 17446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.449800 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "99ad96e5bb05411c9ce9b32b17c907fa"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "97e9fad4d42b8f3a1e9458f2511b44b4"
server_key_iv: "f4aa17a911c30ad6d89e649cb7f76e45"
server_key_version: "encryptionkey@0"
I20260927 03:57:01.450264 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:01.475806 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:01.478134 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:01.478277 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:01.478502 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:01.478799 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:01.478842 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.478871 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:01.478892 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.484516 17446 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40013
I20260927 03:57:01.484565 17576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40013 every 8 connection(s)
I20260927 03:57:01.484887 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:01.487767 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17446
Sep 27 03:57:01 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:01.493912 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.485514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:35059 (local address 127.1.50.126:36067)
0927 03:57:01.485557 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.485559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.485754 (+ 195us) server_negotiation.cc:407] Connection header received
0927 03:57:01.486280 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.486281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.486293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.486351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.487247 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.487730 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.488815 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.489072 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.489127 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.491340 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.491340 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.491341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.491346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.492629 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.492999 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.493000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.493000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.493049 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.493257 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.493258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.493258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.493356 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.493429 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.493661 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.493825 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:57:01.494040 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.485518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:35059)
0927 03:57:01.485645 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.485678 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.486188 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.486415 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.486419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.486975 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:01.487161 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.487166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.487811 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.487812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.488737 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.488741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.489142 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.489143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.489253 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.489720 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:01.489727 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:01.491255 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:01.492697 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.492703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.492705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.492933 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.493100 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:01.493101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:01.493101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:01.493206 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:01.493407 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:01.493413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:01.493548 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.493634 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.493799 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260927 03:57:01.494410 17577 heartbeater.cc:347] Connected to a master server at 127.1.50.126:36067
I20260927 03:57:01.494501 17577 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:01.494683 17577 heartbeater.cc:511] Master 127.1.50.126:36067 requested a full tablet report, sending...
I20260927 03:57:01.494946 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.488267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.1.50.126:36067)
0927 03:57:01.488390 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.488392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.488464 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:57:01.488534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.488535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.488546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.488612 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.489221 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.489873 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.490967 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.491166 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.491216 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.492449 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.492450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.492451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.492457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.493859 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.494249 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.494250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.494250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.494304 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.494468 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.494469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.494469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.494589 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.494726 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.494774 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.494860 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260927 03:57:01.495079 17114 ts_manager.cc:194] Registered new tserver with Master: 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013)
I20260927 03:57:01.495525 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:35059
I20260927 03:57:01.495563 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:01.498139 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.495861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.1.50.126:36067)
0927 03:57:01.495898 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.495900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.495997 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:57:01.496064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.496065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.496092 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.496146 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:01.496258 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.496761 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.497458 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.497702 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.497753 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.497890 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:01.497934 (+ 44us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 03:57:01.498083 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36414: token expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:57:01.498190 17581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36414: token expired
W20260927 03:57:01.498277 17040 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.50.126:36067)
I20260927 03:57:01.503868 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.498513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.1.50.126:36067)
0927 03:57:01.498546 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.498547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.498554 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:57:01.498603 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.498604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.498613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.498667 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:01.499086 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.499530 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.500187 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.500383 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.500434 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.501544 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:01.501545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:01.501545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:01.501551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:01.502865 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.503154 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.503155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.503156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.503208 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:01.503368 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:01.503369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:01.503369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:01.503466 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:01.503551 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.503642 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.503771 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:57:01.505019 17114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:01.506727 17114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:01.508806 17225 tablet_service.cc:1467] Processing CreateTablet for tablet a73398a8aba74f6c9d2516ab63828d4f (DEFAULT_TABLE table=test-table [id=c87afb2a17fd43919099e44d9407aa5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:01.509006 17225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73398a8aba74f6c9d2516ab63828d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:01.509020 17376 tablet_service.cc:1467] Processing CreateTablet for tablet a73398a8aba74f6c9d2516ab63828d4f (DEFAULT_TABLE table=test-table [id=c87afb2a17fd43919099e44d9407aa5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:01.509219 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73398a8aba74f6c9d2516ab63828d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:01.510591 17194 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Bootstrap starting.
I20260927 03:57:01.510612 17333 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Bootstrap starting.
I20260927 03:57:01.510882 17194 tablet_bootstrap.cc:654] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:01.510991 17333 tablet_bootstrap.cc:654] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:01.511354 17194 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: No bootstrap required, opened a new log
I20260927 03:57:01.511400 17194 ts_tablet_manager.cc:1397] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.511520 17333 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: No bootstrap required, opened a new log
I20260927 03:57:01.511559 17333 ts_tablet_manager.cc:1397] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:01.511683 17333 raft_consensus.cc:348] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.511711 17194 raft_consensus.cc:348] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.511750 17333 raft_consensus.cc:374] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:01.511792 17333 raft_consensus.cc:729] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9fe502c81b462ea85f8ad72c7a950a, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.511782 17194 raft_consensus.cc:374] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:01.511823 17194 raft_consensus.cc:729] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0293997e9e75475da428172e476c0828, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.511839 17333 consensus_queue.cc:260] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.511868 17194 consensus_queue.cc:260] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.511924 17333 ts_tablet_manager.cc:1428] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:01.511950 17194 ts_tablet_manager.cc:1428] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:01.511993 17443 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
I20260927 03:57:01.512789 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.508544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40013 (local address 127.0.0.1:40994)
0927 03:57:01.508582 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.508592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.508635 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.509039 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.509040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.509041 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:01.509090 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.509091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.510469 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.510470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.511644 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.511647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.512526 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.512527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.512590 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.512609 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.512666 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.512716 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:57:01.512811 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.508660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.1.50.67:40013)
0927 03:57:01.508864 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.508869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.508880 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:01.508917 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.508920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.508934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.509021 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:01.509152 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.510382 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.511732 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.512454 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.512554 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.512621 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.512682 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.512730 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260927 03:57:01.513422 17511 tablet_service.cc:1467] Processing CreateTablet for tablet a73398a8aba74f6c9d2516ab63828d4f (DEFAULT_TABLE table=test-table [id=c87afb2a17fd43919099e44d9407aa5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:01.513689 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73398a8aba74f6c9d2516ab63828d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:01.516402 17586 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Bootstrap starting.
I20260927 03:57:01.517102 17586 tablet_bootstrap.cc:654] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:01.517616 17586 log.cc:824] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:01.518410 17586 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: No bootstrap required, opened a new log
I20260927 03:57:01.518472 17586 ts_tablet_manager.cc:1397] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:01.519829 17586 raft_consensus.cc:348] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.519968 17586 raft_consensus.cc:374] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:01.519999 17586 raft_consensus.cc:729] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ad96e5bb05411c9ce9b32b17c907fa, State: Initialized, Role: FOLLOWER
I20260927 03:57:01.520110 17586 consensus_queue.cc:260] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.520358 17586 ts_tablet_manager.cc:1428] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:01.520395 17577 heartbeater.cc:503] Master 127.1.50.126:36067 was elected leader, sending a full tablet report...
W20260927 03:57:01.585574 17444 tablet.cc:2369] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:01.608739 17588 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:01.608888 17588 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.609244 17588 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831), 0293997e9e75475da428172e476c0828 (127.1.50.65:41605)
I20260927 03:57:01.612560 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.609529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:37669 (local address 127.1.50.66:35831)
0927 03:57:01.609579 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.609581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.609592 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:01.609624 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.609625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.609654 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.609748 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:01.609966 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.610578 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.611605 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.612147 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.612207 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.612267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.612443 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.612487 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:57:01.612552 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.609437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.1.50.67:37669)
0927 03:57:01.609488 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.609499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.609540 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.609777 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.609779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.609781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:01.609856 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.609857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.610668 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.610669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.611505 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.611508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.612243 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.612244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.612341 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.612346 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.612420 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.612467 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:57:01.612937 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.609533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:51871 (local address 127.1.50.65:41605)
0927 03:57:01.609579 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.609581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.609817 (+ 236us) server_negotiation.cc:407] Connection header received
0927 03:57:01.609852 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.609854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.609879 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.609944 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:01.610185 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.610784 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.611829 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.612523 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.612595 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.612678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.612806 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.612864 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:57:01.613055 17396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73398a8aba74f6c9d2516ab63828d4f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9fe502c81b462ea85f8ad72c7a950a" is_pre_election: true
I20260927 03:57:01.613171 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.609446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.1.50.67:51871)
0927 03:57:01.609727 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.609739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:01.609801 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:01.609974 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:01.609975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:01.609978 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:01.610076 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.610078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.610888 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.610889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.611662 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:01.611664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.612624 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.612625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:01.612703 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.612708 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:01.613053 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 03:57:01.613104 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":52,"thread_start_us":92,"threads_started":1}
I20260927 03:57:01.613261 17396 raft_consensus.cc:2454] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 0.
I20260927 03:57:01.613497 17254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73398a8aba74f6c9d2516ab63828d4f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0293997e9e75475da428172e476c0828" is_pre_election: true
I20260927 03:57:01.613534 17465 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a; no voters:
I20260927 03:57:01.613637 17254 raft_consensus.cc:2454] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 0.
I20260927 03:57:01.613672 17588 raft_consensus.cc:2781] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:01.613727 17588 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:01.613752 17588 raft_consensus.cc:3037] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:01.614555 17588 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.614705 17588 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 election: Requested vote from peers db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831), 0293997e9e75475da428172e476c0828 (127.1.50.65:41605)
I20260927 03:57:01.614883 17396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73398a8aba74f6c9d2516ab63828d4f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9fe502c81b462ea85f8ad72c7a950a"
I20260927 03:57:01.614913 17254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73398a8aba74f6c9d2516ab63828d4f" candidate_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0293997e9e75475da428172e476c0828"
I20260927 03:57:01.614964 17396 raft_consensus.cc:3037] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:01.614986 17254 raft_consensus.cc:3037] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:01.615639 17396 raft_consensus.cc:2454] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 1.
I20260927 03:57:01.615742 17254 raft_consensus.cc:2454] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ad96e5bb05411c9ce9b32b17c907fa in term 1.
I20260927 03:57:01.615803 17465 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a; no voters:
I20260927 03:57:01.615902 17588 raft_consensus.cc:2781] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:01.616074 17588 raft_consensus.cc:686] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 LEADER]: Becoming Leader. State: Replica: 99ad96e5bb05411c9ce9b32b17c907fa, State: Running, Role: LEADER
I20260927 03:57:01.616189 17588 consensus_queue.cc:237] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:01.617146 17114 catalog_manager.cc:5495] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67). New cstate: current_term: 1 leader_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } health_report { overall_health: UNKNOWN } } }
I20260927 03:57:01.629835 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.627512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.1.50.67:40013)
0927 03:57:01.627566 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:01.627568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:01.627576 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:01.627644 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:01.627645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:01.627663 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:01.627721 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:01.627847 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.628339 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.629077 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:01.629301 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:01.629352 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:01.629444 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:01.629564 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:01.629636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:01.629730 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:57:01.629749 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:57:01.631749 17254 raft_consensus.cc:1261] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Refusing update from remote peer 99ad96e5bb05411c9ce9b32b17c907fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:01.631748 17396 raft_consensus.cc:1261] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Refusing update from remote peer 99ad96e5bb05411c9ce9b32b17c907fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:01.632061 17588 consensus_queue.cc:1035] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:01.632205 17590 consensus_queue.cc:1035] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:01.634153 17592 mvcc.cc:205] Tried to move back new op lower bound from 7333811903000489984 to 7333811902940647424. Current Snapshot: MvccSnapshot[applied={T|T < 7333811903000489984}]
I20260927 03:57:01.634706 17594 mvcc.cc:205] Tried to move back new op lower bound from 7333811903000489984 to 7333811902940647424. Current Snapshot: MvccSnapshot[applied={T|T < 7333811903000489984 or (T in {7333811903000489984})}]
I20260927 03:57:01.634965 17593 mvcc.cc:205] Tried to move back new op lower bound from 7333811903000489984 to 7333811902940647424. Current Snapshot: MvccSnapshot[applied={T|T < 7333811903000489984}]
I20260927 03:57:01.636394 17113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:01.636488 17113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:01.637349 17113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Sending DeleteTablet for 3 replicas of tablet a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:01.637656 17225 tablet_service.cc:1514] Processing DeleteTablet for tablet a73398a8aba74f6c9d2516ab63828d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:01 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:40492
I20260927 03:57:01.637753 17311 tablet_replica.cc:330] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: stopping tablet replica
I20260927 03:57:01.637686 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17152
I20260927 03:57:01.637799 17311 raft_consensus.cc:2229] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:01.637861 17311 raft_consensus.cc:2258] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:01.637849 17511 tablet_service.cc:1514] Processing DeleteTablet for tablet a73398a8aba74f6c9d2516ab63828d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:01 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:40994
I20260927 03:57:01.638062 17376 tablet_service.cc:1514] Processing DeleteTablet for tablet a73398a8aba74f6c9d2516ab63828d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:01 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:43996
I20260927 03:57:01.638274 17602 tablet_replica.cc:330] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: stopping tablet replica
I20260927 03:57:01.638319 17448 tablet_replica.cc:330] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: stopping tablet replica
I20260927 03:57:01.638361 17602 raft_consensus.cc:2229] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:57:01.638376 17448 raft_consensus.cc:2229] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:01.638422 17448 raft_consensus.cc:2258] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:01.638476 17602 raft_consensus.cc:2258] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:01.638911 17448 ts_tablet_manager.cc:1905] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:01.638931 17602 ts_tablet_manager.cc:1905] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:01.640635 17448 ts_tablet_manager.cc:1918] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:01.640691 17448 log.cc:1196] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:01.640651 17602 ts_tablet_manager.cc:1918] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:01.640766 17602 log.cc:1196] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:01.640921 17448 ts_tablet_manager.cc:1939] T a73398a8aba74f6c9d2516ab63828d4f P db9fe502c81b462ea85f8ad72c7a950a: Deleting consensus metadata
I20260927 03:57:01.641068 17602 ts_tablet_manager.cc:1939] T a73398a8aba74f6c9d2516ab63828d4f P 99ad96e5bb05411c9ce9b32b17c907fa: Deleting consensus metadata
I20260927 03:57:01.641527 17099 catalog_manager.cc:4841] TS db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): tablet a73398a8aba74f6c9d2516ab63828d4f (table test-table [id=c87afb2a17fd43919099e44d9407aa5a]) successfully deleted
I20260927 03:57:01.641623 17099 catalog_manager.cc:4841] TS 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): tablet a73398a8aba74f6c9d2516ab63828d4f (table test-table [id=c87afb2a17fd43919099e44d9407aa5a]) successfully deleted
W20260927 03:57:01.644944 17100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:41605 (error 108)
I20260927 03:57:01.645182 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17307
I20260927 03:57:01.645498 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:01.645373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41605 (local address 127.0.0.1:40502)
0927 03:57:01.645422 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:01.645455 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:57:01.645694 17100 catalog_manager.cc:4584] TS 0293997e9e75475da428172e476c0828 (127.1.50.65:41605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a73398a8aba74f6c9d2516ab63828d4f: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41605: connect: Connection refused (error 111)
I20260927 03:57:01.650384 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17446
I20260927 03:57:01.654786 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17081
I20260927 03:57:21.659778 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--webserver_interface=127.1.50.126
--webserver_port=38203
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:21.745162 17604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:21.745431 17604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:21.745455 17604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:21.746974 17604 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:57:21.747048 17604 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:57:21.747061 17604 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:57:21.747072 17604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:21.747084 17604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:21.747097 17604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:21.747108 17604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:21.747119 17604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:21.748694 17604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=38203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:21.748921 17604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:21.749099 17604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:21.754627 17609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:21.754673 17610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:21.754659 17612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:21.754681 17604 server_base.cc:1034] running on GCE node
I20260927 03:57:21.755170 17604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:21.755487 17604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:21.756682 17604 hybrid_clock.cc:648] HybridClock initialized: now 1790481441756664 us; error 35 us; skew 500 ppm
Sep 27 03:57:21 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481441, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:21.759545 17604 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:57:21.760113 17604 webserver.cc:460] Webserver started at http://127.1.50.126:38203/ using document root <none> and password file <none>
I20260927 03:57:21.760322 17604 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:21.760373 17604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:21.761895 17604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:21.762755 17619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:21.763082 17604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:21.763186 17604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "4b10dc4693714e3fb94d1ee8bd8a2008"
format_stamp: "Formatted at 2026-09-27 03:56:39 on dist-test-slave-96fg"
server_key: "e2ef70bc68aa0aed838285cd09185cb9"
server_key_iv: "2d71e24bbde301ff3ef86cd215f0f725"
server_key_version: "encryptionkey@0"
I20260927 03:57:21.763581 17604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:21.773249 17604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:21.775385 17604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:21.775518 17604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:21.779744 17604 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:36067
I20260927 03:57:21.779815 17671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:36067 every 8 connection(s)
I20260927 03:57:21.780184 17604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:21.782097 17672 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:57:21.782897 17672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap starting.
I20260927 03:57:21.786082 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17604
I20260927 03:57:21.786504 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:41605
--local_ip_for_outbound_sockets=127.1.50.65
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=41167
--webserver_interface=127.1.50.65
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:57:21.793799 17672 log.cc:824] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:21.797071 17672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:57:21.797421 17672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Bootstrap complete.
I20260927 03:57:21.799817 17672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:21.800081 17672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Initialized, Role: FOLLOWER
I20260927 03:57:21.800267 17672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:21.800343 17672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:21.800385 17672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:21.800436 17672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:57:21.801673 17672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:21.801786 17672 leader_election.cc:302] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b10dc4693714e3fb94d1ee8bd8a2008; no voters:
I20260927 03:57:21.801973 17672 leader_election.cc:288] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:57:21.802064 17676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:57:21.802204 17676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [term 3 LEADER]: Becoming Leader. State: Replica: 4b10dc4693714e3fb94d1ee8bd8a2008, State: Running, Role: LEADER
I20260927 03:57:21.802332 17672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:21.802307 17676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } }
I20260927 03:57:21.802801 17677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:57:21.802907 17677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:21.803064 17678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b10dc4693714e3fb94d1ee8bd8a2008. Latest consensus state: current_term: 3 leader_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b10dc4693714e3fb94d1ee8bd8a2008" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 36067 } } }
I20260927 03:57:21.803129 17678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:21.803234 17684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:21.804189 17684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:21.804297 17684 catalog_manager.cc:1259] Loaded cluster ID: 13d778b610cb4703811764fced951627
I20260927 03:57:21.804335 17684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:21.805514 17684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:21.805725 17684 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Loaded TSK: 0
I20260927 03:57:21.805778 17684 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b10dc4693714e3fb94d1ee8bd8a2008: Loaded TSK: 1
F20260927 03:57:21.806057 17684 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fea4dbbd9ad google::LogMessage::Fail() at ??:0
@ 0x7fea4dbc088a google::LogMessage::SendToLog() at ??:0
@ 0x7fea4dbbd4de google::LogMessage::Flush() at ??:0
@ 0x7fea4dbbea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fea5032a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fea50327d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fea55b1b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fea55b1b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fea55b1b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fea55ae1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fea55aebad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fea4fa8b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fea4fa80954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fea508e46db start_thread at ??:0
@ 0x7fea4c6bb61f clone at ??:0
W20260927 03:57:21.901777 17674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:21.902143 17674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:21.902187 17674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:21.904213 17674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:21.904314 17674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:21.904371 17674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:21.907043 17674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:41605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=41167
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:21.907379 17674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:21.907670 17674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:21.914193 17714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:21.914193 17713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:21.914471 17674 server_base.cc:1034] running on GCE node
W20260927 03:57:21.914462 17716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:21.914837 17674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:21.915174 17674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:21.916361 17674 hybrid_clock.cc:648] HybridClock initialized: now 1790481441916346 us; error 36 us; skew 500 ppm
Sep 27 03:57:21 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481441, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:21.919834 17674 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:57:21.920421 17674 webserver.cc:460] Webserver started at http://127.1.50.65:41167/ using document root <none> and password file <none>
I20260927 03:57:21.920701 17674 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:21.920760 17674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:21.922390 17674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:21.923225 17723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:21.923499 17674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:21.923600 17674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "0293997e9e75475da428172e476c0828"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "e1ffab299f5806f5d4a90831510197b3"
server_key_iv: "e957f4d9de536a653f4ad072946a56ee"
server_key_version: "encryptionkey@0"
I20260927 03:57:21.924018 17674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:21.949395 17674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:21.952155 17674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:21.952320 17674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:21.952610 17674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:21.953159 17730 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:57:21.955469 17674 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:57:21.955545 17674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260927 03:57:21.955590 17674 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:57:21.956467 17674 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:57:21.956506 17674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:21.956629 17730 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Bootstrap starting.
I20260927 03:57:21.964682 17674 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:41605
I20260927 03:57:21.964880 17837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:41605 every 8 connection(s)
I20260927 03:57:21.965224 17674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:21.967891 17730 log.cc:824] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:21.971029 17730 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:57:21.971388 17730 tablet_bootstrap.cc:492] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Bootstrap complete.
I20260927 03:57:21.972489 17730 ts_tablet_manager.cc:1397] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260927 03:57:21.973834 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17674
I20260927 03:57:21.973925 17730 raft_consensus.cc:348] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:21.974385 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:35831
--local_ip_for_outbound_sockets=127.1.50.66
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=37299
--webserver_interface=127.1.50.66
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:57:21.974319 17730 raft_consensus.cc:729] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0293997e9e75475da428172e476c0828, State: Initialized, Role: FOLLOWER
I20260927 03:57:21.974660 17730 consensus_queue.cc:260] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:21.975075 17730 ts_tablet_manager.cc:1428] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260927 03:57:22.030004 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:57:21.966090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:55797)
0927 03:57:21.966272 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:21.966321 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:21.966874 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:22.029779 (+ 62905us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260927 03:57:22.030359 17838 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:36067 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:36067: Client connection negotiation failed: client connection to 127.1.50.126:36067: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:57:22.030819 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.030671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:42531)
0927 03:57:22.030723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.030789 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:57:22.031272 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.031184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:45273)
0927 03:57:22.031232 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.031257 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:57:22.031385 17838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:57:22.068252 17843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:22.068512 17843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:22.068538 17843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:22.069821 17843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:22.069868 17843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:22.069895 17843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:22.071789 17843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:35831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=37299
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:22.072083 17843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:22.072314 17843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:22.078250 17849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:22.078250 17848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:22.078439 17843 server_base.cc:1034] running on GCE node
W20260927 03:57:22.078382 17851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:22.078846 17843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:22.079149 17843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:22.080333 17843 hybrid_clock.cc:648] HybridClock initialized: now 1790481442080305 us; error 48 us; skew 500 ppm
Sep 27 03:57:22 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481442, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:22.083364 17843 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:57:22.083962 17843 webserver.cc:460] Webserver started at http://127.1.50.66:37299/ using document root <none> and password file <none>
I20260927 03:57:22.084200 17843 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:22.084254 17843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:22.086004 17843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:22.086786 17858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.086994 17843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:22.087087 17843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "db9fe502c81b462ea85f8ad72c7a950a"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "854d1b1284bf122588643c0876862159"
server_key_iv: "78206891b26346ebf6d101c496a2730b"
server_key_version: "encryptionkey@0"
I20260927 03:57:22.087512 17843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:22.103207 17843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:22.105496 17843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:22.105679 17843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:22.105937 17843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:22.106287 17843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:22.106323 17843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.106353 17843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:22.106370 17843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.113447 17843 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:35831
I20260927 03:57:22.113507 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:35831 every 8 connection(s)
I20260927 03:57:22.113888 17843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:57:22.115237 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.114724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:58617)
0927 03:57:22.114978 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.115061 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":163,"threads_started":1}
W20260927 03:57:22.115721 17972 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:36067 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:36067: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
I20260927 03:57:22.116041 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.115944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:43949)
0927 03:57:22.115993 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.116022 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:57:22.116511 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.116320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:35533)
0927 03:57:22.116404 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.116440 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 03:57:22.116693 17972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:57:22.121263 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17843
I20260927 03:57:22.121601 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:40013
--local_ip_for_outbound_sockets=127.1.50.67
--tserver_master_addrs=127.1.50.126:36067
--webserver_port=43141
--webserver_interface=127.1.50.67
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:22.210526 17975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:22.210788 17975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:22.210815 17975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:22.212097 17975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:22.212147 17975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:22.212175 17975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:22.213958 17975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:40013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=43141
--tserver_master_addrs=127.1.50.126:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:22.214242 17975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:22.214481 17975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:22.220188 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:22.220188 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:22.220320 17975 server_base.cc:1034] running on GCE node
W20260927 03:57:22.220284 17983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:22.220763 17975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:22.221058 17975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:22.222222 17975 hybrid_clock.cc:648] HybridClock initialized: now 1790481442222198 us; error 44 us; skew 500 ppm
Sep 27 03:57:22 dist-test-slave-96fg krb5kdc[16533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481442, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:22.225011 17975 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:57:22.225540 17975 webserver.cc:460] Webserver started at http://127.1.50.67:43141/ using document root <none> and password file <none>
I20260927 03:57:22.225788 17975 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:22.225841 17975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:22.227643 17975 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:22.228500 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.228691 17975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:22.228782 17975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "99ad96e5bb05411c9ce9b32b17c907fa"
format_stamp: "Formatted at 2026-09-27 03:56:40 on dist-test-slave-96fg"
server_key: "97e9fad4d42b8f3a1e9458f2511b44b4"
server_key_iv: "f4aa17a911c30ad6d89e649cb7f76e45"
server_key_version: "encryptionkey@0"
I20260927 03:57:22.229218 17975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:22.251453 17975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:22.253587 17975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:22.253724 17975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:22.253970 17975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:22.254334 17975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:22.254374 17975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.254396 17975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:22.254408 17975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:22.262042 17975 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:40013
I20260927 03:57:22.262169 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:40013 every 8 connection(s)
I20260927 03:57:22.262522 17975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:22.263868 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.263404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:58069)
0927 03:57:22.263611 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.263702 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260927 03:57:22.264315 18104 heartbeater.cc:644] Failed to heartbeat to 127.1.50.126:36067 (0 consecutive failures): Network error: Failed to ping master at 127.1.50.126:36067: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
I20260927 03:57:22.264612 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.264516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:50393)
0927 03:57:22.264565 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.264593 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:57:22.264876 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:22.264797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:44759)
0927 03:57:22.264841 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:22.264862 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:57:22.265007 18104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:57:22.267753 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 17975
I20260927 03:57:23.032232 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.031928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:46201)
0927 03:57:23.032084 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:23.032131 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260927 03:57:23.117470 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.117161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:33767)
0927 03:57:23.117322 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:23.117389 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260927 03:57:23.265765 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.265426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:47621)
0927 03:57:23.265598 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:23.265678 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260927 03:57:23.625494 18111 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:23.625656 18111 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:23.626008 18111 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
Sep 27 03:57:23 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481441, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.66@KRBTEST.COM
Sep 27 03:57:23 dist-test-slave-96fg krb5kdc[16533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481441, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.67@KRBTEST.COM
I20260927 03:57:23.637709 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.626455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:45585 (local address 127.1.50.67:40013)
0927 03:57:23.626642 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:23.626658 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:23.626698 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:57:23.627249 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:23.627252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:23.627301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:23.627453 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:23.628357 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.629279 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.630439 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.630898 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.631012 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:23.634289 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:23.634295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:23.634295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:23.634307 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:23.636008 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636475 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:23.636478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:23.636479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:23.636576 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636769 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:23.636770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:23.636771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:23.637041 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:23.637119 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:23.637429 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:57:23.637576 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":55,"threads_started":1}
I20260927 03:57:23.637725 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.626649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44363 (local address 127.1.50.66:35831)
0927 03:57:23.626791 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:23.626803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:23.626827 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:57:23.627405 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:23.627409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:23.627470 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:23.627629 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:23.628376 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.629310 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.630446 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.630880 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.630983 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:23.633944 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:23.633952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:23.633952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:23.633969 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:23.635885 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636397 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:23.636401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:23.636402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:23.636513 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636775 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:23.636776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:23.636776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:23.637042 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:23.637122 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:23.637446 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 03:57:23.637588 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260927 03:57:23.637712 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.626306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:40013 (local address 127.1.50.65:45585)
0927 03:57:23.626453 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:23.626465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:23.626546 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:23.627512 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:23.627516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:23.627978 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:23.628248 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:23.628256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.629386 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.629387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:23.630351 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:23.630355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.630989 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.630990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:23.631097 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:23.631845 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:23.631850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:23.634167 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:23.636115 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:23.636117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:23.636385 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:23.636646 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:23.636647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:23.636705 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:23.637117 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:23.637129 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:23.637262 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:57:23.637361 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:57:23.637605 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260927 03:57:23.638165 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:23.626525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35831 (local address 127.1.50.65:44363)
0927 03:57:23.626620 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:23.626631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:23.626693 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:23.627626 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:23.627644 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:23.627982 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:23.628250 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:23.628262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.629421 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.629422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:23.630354 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:23.630357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:23.630973 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:23.630974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:23.631100 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:23.631769 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:23.631780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:23.633743 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:23.635993 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:23.636003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:23.636309 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:23.636581 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:23.636581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:23.636582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:23.636706 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:23.637116 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:23.637123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:23.637260 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:57:23.637359 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:57:23.637692 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":303,"thread_start_us":37,"threads_started":1}
W20260927 03:57:23.638734 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
W20260927 03:57:23.638994 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:23.639062 17725 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0293997e9e75475da428172e476c0828; no voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a
I20260927 03:57:23.639187 18111 raft_consensus.cc:2726] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:57:24.033013 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:24.032829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:34365)
0927 03:57:24.032913 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:24.032969 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 03:57:24.118247 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:24.118002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:39247)
0927 03:57:24.118140 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:24.118195 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260927 03:57:24.266650 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:24.266379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:56173)
0927 03:57:24.266563 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:24.266607 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":60,"threads_started":1}
I20260927 03:57:25.033950 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:25.033626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:55109)
0927 03:57:25.033810 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:25.033882 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260927 03:57:25.119213 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:25.118846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:41641)
0927 03:57:25.119082 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:25.119147 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260927 03:57:25.267449 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:25.267176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:49045)
0927 03:57:25.267334 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:25.267391 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260927 03:57:25.518406 18121 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:25.518549 18121 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:25.518721 18121 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
W20260927 03:57:25.519308 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
W20260927 03:57:25.519407 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:25.519477 17725 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0293997e9e75475da428172e476c0828; no voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a
I20260927 03:57:25.519584 18121 raft_consensus.cc:2726] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:57:26.034821 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:26.034462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:40493)
0927 03:57:26.034686 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:26.034757 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260927 03:57:26.120023 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:26.119721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:54801)
0927 03:57:26.119890 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:26.119960 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260927 03:57:26.268369 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:26.268031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:43673)
0927 03:57:26.268258 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:26.268302 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":133,"threads_started":1}
I20260927 03:57:27.035753 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:27.035424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:52427)
0927 03:57:27.035616 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:27.035687 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260927 03:57:27.120901 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:27.120596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:44301)
0927 03:57:27.120767 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:27.120835 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260927 03:57:27.269343 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:27.268999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:42827)
0927 03:57:27.269196 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:27.269280 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260927 03:57:28.036883 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:28.036480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:51603)
0927 03:57:28.036765 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:28.036831 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":161,"threads_started":1}
I20260927 03:57:28.121881 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:28.121557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:34789)
0927 03:57:28.121744 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:28.121814 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260927 03:57:28.174243 18130 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:28.174383 18130 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:28.174644 18130 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
W20260927 03:57:28.175107 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
W20260927 03:57:28.175210 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:28.175251 17725 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0293997e9e75475da428172e476c0828; no voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a
I20260927 03:57:28.175335 18130 raft_consensus.cc:2726] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:57:28.270349 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:28.269978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:37003)
0927 03:57:28.270202 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:28.270296 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":140,"threads_started":1}
I20260927 03:57:29.037917 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:29.037575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:45375)
0927 03:57:29.037808 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:29.037854 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":157,"threads_started":1}
I20260927 03:57:29.122833 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:29.122541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:41223)
0927 03:57:29.122724 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:29.122783 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260927 03:57:29.271301 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:29.270972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:48815)
0927 03:57:29.271180 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:29.271245 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260927 03:57:30.038830 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:30.038527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:52293)
0927 03:57:30.038685 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:30.038727 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260927 03:57:30.123663 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:30.123369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:58519)
0927 03:57:30.123556 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:30.123612 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260927 03:57:30.272327 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:30.271959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:35009)
0927 03:57:30.272200 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:30.272263 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260927 03:57:31.039698 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:31.039351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:32821)
0927 03:57:31.039554 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:31.039627 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260927 03:57:31.124569 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:31.124275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:45165)
0927 03:57:31.124453 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:31.124521 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260927 03:57:31.273352 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:31.273012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:38399)
0927 03:57:31.273239 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:31.273303 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260927 03:57:32.040619 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:32.040375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:36421)
0927 03:57:32.040532 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:32.040571 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260927 03:57:32.125562 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:32.125259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:46075)
0927 03:57:32.125432 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:32.125484 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260927 03:57:32.165150 18143 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:32.165263 18143 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:32.165448 18143 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
W20260927 03:57:32.166119 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
W20260927 03:57:32.166239 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:32.166254 17725 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0293997e9e75475da428172e476c0828; no voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a
I20260927 03:57:32.166352 18143 raft_consensus.cc:2726] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:57:32.274431 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:32.274102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:48981)
0927 03:57:32.274292 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:32.274363 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
I20260927 03:57:33.041501 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:33.041240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:33465)
0927 03:57:33.041412 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:33.041453 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260927 03:57:33.126650 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:33.126293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:35257)
0927 03:57:33.126481 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:33.126552 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260927 03:57:33.275424 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:33.275096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:60953)
0927 03:57:33.275289 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:33.275357 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260927 03:57:34.042423 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:34.042205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:57649)
0927 03:57:34.042337 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:34.042375 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260927 03:57:34.127601 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:34.127264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:60569)
0927 03:57:34.127467 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:34.127534 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260927 03:57:34.276449 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:34.276110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:43275)
0927 03:57:34.276320 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:34.276386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260927 03:57:35.043256 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:35.042985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:43329)
0927 03:57:35.043149 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:35.043208 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260927 03:57:35.128448 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:35.128170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:52705)
0927 03:57:35.128337 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:35.128397 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260927 03:57:35.277415 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:35.277108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:33101)
0927 03:57:35.277303 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:35.277364 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
I20260927 03:57:35.934365 18154 raft_consensus.cc:482] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:35.934489 18154 raft_consensus.cc:504] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ad96e5bb05411c9ce9b32b17c907fa" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 40013 } } peers { permanent_uuid: "db9fe502c81b462ea85f8ad72c7a950a" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35831 } } peers { permanent_uuid: "0293997e9e75475da428172e476c0828" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41605 } }
I20260927 03:57:35.934652 18154 leader_election.cc:288] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013), db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831)
W20260927 03:57:35.935362 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99ad96e5bb05411c9ce9b32b17c907fa (127.1.50.67:40013): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
W20260927 03:57:35.935477 17725 leader_election.cc:341] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db9fe502c81b462ea85f8ad72c7a950a (127.1.50.66:35831): Not found: Tablet not found: a73398a8aba74f6c9d2516ab63828d4f
I20260927 03:57:35.935503 17725 leader_election.cc:302] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0293997e9e75475da428172e476c0828; no voters: 99ad96e5bb05411c9ce9b32b17c907fa, db9fe502c81b462ea85f8ad72c7a950a
I20260927 03:57:35.935619 18154 raft_consensus.cc:2726] T a73398a8aba74f6c9d2516ab63828d4f P 0293997e9e75475da428172e476c0828 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:57:36.044200 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:36.043907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:57841)
0927 03:57:36.044099 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:36.044151 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260927 03:57:36.129436 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:36.129047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:39587)
0927 03:57:36.129283 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:36.129359 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260927 03:57:36.278465 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:36.278122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:53921)
0927 03:57:36.278331 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:36.278396 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260927 03:57:37.045217 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:37.044961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.65:45847)
0927 03:57:37.045105 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:37.045151 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260927 03:57:37.130720 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:37.130346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.66:51687)
0927 03:57:37.130564 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:37.130639 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260927 03:57:37.270462 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17674
I20260927 03:57:37.275370 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17843
I20260927 03:57:37.279346 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:37.279097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:36067 (local address 127.1.50.67:45069)
0927 03:57:37.279247 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:37.279291 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.126:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260927 03:57:37.279994 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 17975
2026-09-27T03:57:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58559 ms)
[----------] 30 tests from SecurityITest (234740 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:37 dist-test-slave-96fg krb5kdc[18163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:37 dist-test-slave-96fg krb5kdc[18163](info): set up 2 sockets
Sep 27 03:57:37 dist-test-slave-96fg krb5kdc[18163](info): commencing operation
krb5kdc: starting...
W20260927 03:57:38.253619 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:57:38.374894 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37919
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:37919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:38.458676 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:38.458878 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:38.458899 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:38.460024 18176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:38.460057 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:38.460069 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:38.460080 18176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:38.460090 18176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:38.461522 18176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:37919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:38.461733 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:38.461911 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:38.466542 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.466615 18176 server_base.cc:1034] running on GCE node
W20260927 03:57:38.466563 18181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.466542 18182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.467059 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:38.467329 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:38.468518 18176 hybrid_clock.cc:648] HybridClock initialized: now 1790481458468500 us; error 30 us; skew 500 ppm
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:38.471007 18176 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:57:38.471544 18176 webserver.cc:460] Webserver started at http://127.1.50.126:36349/ using document root <none> and password file <none>
I20260927 03:57:38.471745 18176 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:38.471791 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:38.471921 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:38.472951 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "63b6fda396be4a59bb780ec47cc8af84"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "60355f03f507ba8039236e18d9f668c2"
server_key_iv: "00def4d060dfb8c68536816d2d4a4e84"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.473268 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "63b6fda396be4a59bb780ec47cc8af84"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "60355f03f507ba8039236e18d9f668c2"
server_key_iv: "00def4d060dfb8c68536816d2d4a4e84"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.474511 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:38.475229 18191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.475382 18176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:38.475461 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "63b6fda396be4a59bb780ec47cc8af84"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "60355f03f507ba8039236e18d9f668c2"
server_key_iv: "00def4d060dfb8c68536816d2d4a4e84"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.475519 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:38.490379 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:38.492493 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:38.492682 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:38.496417 18176 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:37919
I20260927 03:57:38.496476 18243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:37919 every 8 connection(s)
I20260927 03:57:38.496774 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:38.497597 18244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:38.499852 18244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: Bootstrap starting.
I20260927 03:57:38.500023 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18176
I20260927 03:57:38.500115 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:38.500306 1225 external_mini_cluster.cc:1428] Setting key 4a1f7529df2d90aa13094432f3dc42e8
I20260927 03:57:38.500582 18244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:38.500885 18244 log.cc:824] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:38.501543 18244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: No bootstrap required, opened a new log
I20260927 03:57:38.503741 18244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } }
I20260927 03:57:38.503880 18244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:38.503912 18244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b6fda396be4a59bb780ec47cc8af84, State: Initialized, Role: FOLLOWER
I20260927 03:57:38.504022 18244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } }
I20260927 03:57:38.504093 18244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:38.504132 18244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:38.504174 18244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:38.504842 18244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } }
I20260927 03:57:38.504963 18244 leader_election.cc:302] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63b6fda396be4a59bb780ec47cc8af84; no voters:
I20260927 03:57:38.505129 18244 leader_election.cc:288] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:38.505154 18249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:38.505259 18249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [term 1 LEADER]: Becoming Leader. State: Replica: 63b6fda396be4a59bb780ec47cc8af84, State: Running, Role: LEADER
I20260927 03:57:38.505388 18244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:38.505390 18249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } }
I20260927 03:57:38.505842 18251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63b6fda396be4a59bb780ec47cc8af84. Latest consensus state: current_term: 1 leader_uuid: "63b6fda396be4a59bb780ec47cc8af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } } }
I20260927 03:57:38.505954 18251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:38.506636 18258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:38.507123 18258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:38.507526 18250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63b6fda396be4a59bb780ec47cc8af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b6fda396be4a59bb780ec47cc8af84" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37919 } } }
I20260927 03:57:38.507673 18250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84 [sys.catalog]: This master's current role is: LEADER
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:38.508633 18258 catalog_manager.cc:1347] Generated new cluster ID: a7ae393a8bc14da890c2be01a8b7d6c3
I20260927 03:57:38.508685 18258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:38.511317 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:38.500853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.1.50.126:37919)
0927 03:57:38.501110 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:38.501118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:38.501145 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:57:38.501597 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:38.502097 (+ 500us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:38.502168 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:38.502298 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:38.503037 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.503950 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.505004 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.505570 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.505675 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:38.508484 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:38.508488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:38.508489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:38.508500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:38.509998 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.510315 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.510318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.510318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.510394 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.510564 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.510565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.510566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.510777 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:38.510935 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:38.511016 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 03:57:38.511143 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260927 03:57:38.521240 18258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:38.521974 18258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:38.528014 18258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: Generated new TSK 0
I20260927 03:57:38.528198 18258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:57:38.561336 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:38.641687 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:38.641919 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:38.641954 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:38.643245 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:38.643299 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:38.643337 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:38.644906 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:38.645169 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:38.645385 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:38.651170 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.651152 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.651247 18270 server_base.cc:1034] running on GCE node
W20260927 03:57:38.651170 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.651644 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:38.651930 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:38.653112 18270 hybrid_clock.cc:648] HybridClock initialized: now 1790481458653101 us; error 39 us; skew 500 ppm
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:38.656056 18270 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:57:38.656466 18270 webserver.cc:460] Webserver started at http://127.1.50.65:35501/ using document root <none> and password file <none>
I20260927 03:57:38.656689 18270 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:38.656738 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:38.656859 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:38.657866 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "0f6c9c24651148f8a9c216b986e0d457"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "860c23a457c230e5e675f01c4aaac396"
server_key_iv: "9484736b77c6e88c777e5fdb59e179f7"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.658224 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "0f6c9c24651148f8a9c216b986e0d457"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "860c23a457c230e5e675f01c4aaac396"
server_key_iv: "9484736b77c6e88c777e5fdb59e179f7"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.659479 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:38.660255 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.660463 18270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:38.660552 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "0f6c9c24651148f8a9c216b986e0d457"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "860c23a457c230e5e675f01c4aaac396"
server_key_iv: "9484736b77c6e88c777e5fdb59e179f7"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.660616 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:38.693166 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:38.695299 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:38.695443 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:38.695686 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:38.695977 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:38.696010 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.696039 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:38.696056 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.701676 18270 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:36037
I20260927 03:57:38.701726 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:36037 every 8 connection(s)
I20260927 03:57:38.702056 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:38.706750 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18270
I20260927 03:57:38.706844 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:38.707089 1225 external_mini_cluster.cc:1428] Setting key ac26098e7de81acfcc5fda366080e9bc
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:38.711511 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:38.702735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:60945 (local address 127.1.50.126:37919)
0927 03:57:38.702793 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:38.702795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:38.702989 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:57:38.703452 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:38.703453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:38.703462 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:38.703519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:38.704451 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.705003 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.706142 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.706363 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.706423 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:38.708733 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:38.708734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:38.708735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:38.708742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:38.710135 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.710564 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.710565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.710565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.710624 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.710844 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.710844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.710845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.710955 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:38.711016 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:38.711300 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:57:38.711391 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:57:38.711649 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:38.702699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37919 (local address 127.1.50.65:60945)
0927 03:57:38.702855 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:38.702896 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:38.703376 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:38.703569 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:38.703572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:38.704183 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:38.704376 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:38.704381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.705085 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.705086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:38.705989 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:38.705993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.706449 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.706450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:38.706559 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:38.707121 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:38.707131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:38.708633 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:38.710230 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:38.710236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:38.710238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:38.710464 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:38.710684 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:38.710684 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:38.710685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:38.710785 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:38.711021 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:38.711028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:38.711164 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:57:38.711269 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:57:38.711428 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260927 03:57:38.712129 18399 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37919
I20260927 03:57:38.712219 18399 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:38.712391 18399 heartbeater.cc:511] Master 127.1.50.126:37919 requested a full tablet report, sending...
I20260927 03:57:38.712901 18208 ts_manager.cc:194] Registered new tserver with Master: 0f6c9c24651148f8a9c216b986e0d457 (127.1.50.65:36037)
I20260927 03:57:38.713502 18208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:60945
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:57:38.733410 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:38.816293 18404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:38.816510 18404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:38.816543 18404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:38.817730 18404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:38.817773 18404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:38.817809 18404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:38.819319 18404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:38.819564 18404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:38.819778 18404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:38.824723 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.824729 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.824740 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.824782 18404 server_base.cc:1034] running on GCE node
I20260927 03:57:38.825107 18404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:38.825313 18404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:38.826479 18404 hybrid_clock.cc:648] HybridClock initialized: now 1790481458826463 us; error 29 us; skew 500 ppm
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:38.829176 18404 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:57:38.829627 18404 webserver.cc:460] Webserver started at http://127.1.50.66:39189/ using document root <none> and password file <none>
I20260927 03:57:38.829833 18404 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:38.829880 18404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:38.830003 18404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:38.831055 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "78a3bb58f1b443ca9c43d95f21401ca2"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "5662d485049f98991cdd3dbfc18944d6"
server_key_iv: "ad7f41ea1a6669b6424b8261ad0cc01f"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.831365 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "78a3bb58f1b443ca9c43d95f21401ca2"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "5662d485049f98991cdd3dbfc18944d6"
server_key_iv: "ad7f41ea1a6669b6424b8261ad0cc01f"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.832581 18404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:38.833252 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.833416 18404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:38.833499 18404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "78a3bb58f1b443ca9c43d95f21401ca2"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "5662d485049f98991cdd3dbfc18944d6"
server_key_iv: "ad7f41ea1a6669b6424b8261ad0cc01f"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.833630 18404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:38.846863 18404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:38.849134 18404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:38.849261 18404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:38.849474 18404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:38.849781 18404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:38.849819 18404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.849840 18404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:38.849862 18404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.855360 18404 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:35291
I20260927 03:57:38.855535 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:35291 every 8 connection(s)
I20260927 03:57:38.855715 18404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:57:38.858861 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18404
I20260927 03:57:38.858974 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:38.859154 1225 external_mini_cluster.cc:1428] Setting key 7c48feaf2eb5b2b336f71795eba36efc
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:38.867192 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:38.856544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:58067 (local address 127.1.50.126:37919)
0927 03:57:38.856653 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:38.856655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:38.856847 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:57:38.857440 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:38.857442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:38.857457 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:38.857517 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:38.858683 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.859383 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.860822 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.861075 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.861132 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:38.863625 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:38.863627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:38.863627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:38.863635 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:38.865323 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.865798 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.865799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.865800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.865864 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:38.866196 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:38.866197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:38.866198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:38.866348 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:38.866428 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:38.867002 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 03:57:38.867092 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 03:57:38.867288 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:38.856479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37919 (local address 127.1.50.66:58067)
0927 03:57:38.856733 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:38.856769 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:38.857288 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:38.857580 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:38.857584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:38.858302 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:38.858567 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:38.858574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.859497 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.859498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:38.860727 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:38.860730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:38.861154 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:38.861155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:38.861306 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:38.861854 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:38.861863 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:38.863525 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:38.865442 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:38.865450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:38.865452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:38.865720 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:38.865957 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:38.865958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:38.865959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:38.866128 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:38.866502 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:38.866509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:38.866836 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 03:57:38.866932 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:57:38.867086 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
I20260927 03:57:38.868196 18533 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37919
I20260927 03:57:38.868297 18533 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:38.868445 18533 heartbeater.cc:511] Master 127.1.50.126:37919 requested a full tablet report, sending...
I20260927 03:57:38.868865 18208 ts_manager.cc:194] Registered new tserver with Master: 78a3bb58f1b443ca9c43d95f21401ca2 (127.1.50.66:35291)
I20260927 03:57:38.869295 18208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:58067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:57:38.885792 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:38.966812 18538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:38.967044 18538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:38.967077 18538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:38.968317 18538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:38.968369 18538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:38.968397 18538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:38.969923 18538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:38.970162 18538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:38.970386 18538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:38.975382 18544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.975382 18546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:38.975383 18543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:38.975449 18538 server_base.cc:1034] running on GCE node
I20260927 03:57:38.975834 18538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:38.976073 18538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:38.977228 18538 hybrid_clock.cc:648] HybridClock initialized: now 1790481458977216 us; error 26 us; skew 500 ppm
Sep 27 03:57:38 dist-test-slave-96fg krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:38.979780 18538 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:57:38.980257 18538 webserver.cc:460] Webserver started at http://127.1.50.67:35405/ using document root <none> and password file <none>
I20260927 03:57:38.980484 18538 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:38.980533 18538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:38.980655 18538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:38.981712 18538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "ce6c1a5b5b7445e792232a905cc54ca3"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "4abbb328a894c47f662bacdb551ae04d"
server_key_iv: "784dfd73b30b635feec9273c2799e7a6"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.982098 18538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "ce6c1a5b5b7445e792232a905cc54ca3"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "4abbb328a894c47f662bacdb551ae04d"
server_key_iv: "784dfd73b30b635feec9273c2799e7a6"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.983417 18538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:38.984246 18553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.984479 18538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:38.984572 18538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "ce6c1a5b5b7445e792232a905cc54ca3"
format_stamp: "Formatted at 2026-09-27 03:57:38 on dist-test-slave-96fg"
server_key: "4abbb328a894c47f662bacdb551ae04d"
server_key_iv: "784dfd73b30b635feec9273c2799e7a6"
server_key_version: "encryptionkey@0"
I20260927 03:57:38.984678 18538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:38.996836 18538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:38.999001 18538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:38.999135 18538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:38.999351 18538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:38.999648 18538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:38.999682 18538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:38.999711 18538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:38.999734 18538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.005203 18538 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42917
I20260927 03:57:39.005265 18666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42917 every 8 connection(s)
I20260927 03:57:39.005632 18538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:39.010986 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18538
I20260927 03:57:39.011096 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:39.011265 1225 external_mini_cluster.cc:1428] Setting key 6091990282beee554c0186f17f30ca67
Sep 27 03:57:39 dist-test-slave-96fg krb5kdc[18163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:39.014978 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.006340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:48377 (local address 127.1.50.126:37919)
0927 03:57:39.006381 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.006383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.006593 (+ 210us) server_negotiation.cc:407] Connection header received
0927 03:57:39.007132 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.007133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.007144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.007179 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.008069 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.008563 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.009560 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.009754 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.009816 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.012131 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.012133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.012133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:39.012141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.013465 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:39.013860 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:39.013861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:39.013861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:39.013931 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:39.014224 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:39.014224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:39.014224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:39.014362 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.014422 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.014799 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.014880 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:57:39.015136 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.006327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37919 (local address 127.1.50.67:48377)
0927 03:57:39.006462 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.006505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.007040 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.007237 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.007241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.007777 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:39.007984 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.007988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.008638 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.008639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.009460 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.009463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.009838 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.009839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.009946 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.010462 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:39.010470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:39.012035 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:39.013538 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:39.013544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:39.013547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:39.013797 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:39.014007 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:39.014008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:39.014008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:39.014162 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:39.014461 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:39.014468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:39.014627 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.014755 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.014914 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260927 03:57:39.015681 18667 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37919
I20260927 03:57:39.015787 18667 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:39.015970 18667 heartbeater.cc:511] Master 127.1.50.126:37919 requested a full tablet report, sending...
I20260927 03:57:39.016345 18208 ts_manager.cc:194] Registered new tserver with Master: ce6c1a5b5b7445e792232a905cc54ca3 (127.1.50.67:42917)
I20260927 03:57:39.016801 18208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:48377
I20260927 03:57:39.022912 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:39.030077 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.024314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.1.50.126:37919)
0927 03:57:39.024355 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.024357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.024475 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:57:39.024564 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.024565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.024575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.024622 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.025143 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.025624 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.026364 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.026564 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.026617 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.027685 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.027686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.027686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:39.027693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.029009 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:39.029322 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:39.029323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:39.029323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:39.029380 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:39.029569 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:39.029570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:39.029570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:39.029714 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.029793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.029895 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.029975 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:57:39.031646 18208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:39.033174 18208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:39.039426 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.1.50.67:42917)
0927 03:57:39.035824 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.035828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.035837 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:39.035869 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.035871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.035883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.035953 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.036136 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037058 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.038369 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039056 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039115 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.039176 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.039299 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.039339 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":68,"threads_started":1}
I20260927 03:57:39.039415 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42917 (local address 127.0.0.1:39798)
0927 03:57:39.035609 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.035648 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.035714 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.035968 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.035971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.035973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.036039 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.036040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037157 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.037157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.038279 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.038281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039133 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.039207 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.039220 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.039280 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.039334 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":59,"threads_started":1}
I20260927 03:57:39.040089 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36037 (local address 127.0.0.1:33914)
0927 03:57:39.035629 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.035658 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.035714 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.036290 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.036290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.036292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.036336 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.036337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037695 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.037697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.038817 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.038819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039855 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.039926 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.039930 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.039978 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.040019 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260927 03:57:39.040115 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.1.50.65:36037)
0927 03:57:39.036091 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.036095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.036103 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:39.036137 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.036139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.036152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.036223 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.036396 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037589 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.038901 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039558 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039758 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.039821 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.039994 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.040042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":87,"threads_started":1}
I20260927 03:57:39.040397 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.1.50.66:35291)
0927 03:57:39.035795 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.035800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.036425 (+ 625us) server_negotiation.cc:407] Connection header received
0927 03:57:39.036490 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.036492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.036506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.036582 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.036676 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037680 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.038767 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039820 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039888 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.039938 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.040277 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.040320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260927 03:57:39.040347 18601 tablet_service.cc:1467] Processing CreateTablet for tablet ed66a47f768b4edd975f420afed071cf (DEFAULT_TABLE table=test-table [id=80ce18ea231b46a8b8a1d48652a03ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.040604 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.035504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35291 (local address 127.0.0.1:54192)
0927 03:57:39.036364 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.036373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.036428 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.036581 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.036582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.036584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.036620 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.036622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.037756 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.037757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.038695 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.038697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.039911 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.039912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.040032 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.040039 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.040102 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.040518 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":92,"threads_started":1}
I20260927 03:57:39.040748 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed66a47f768b4edd975f420afed071cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.041428 18467 tablet_service.cc:1467] Processing CreateTablet for tablet ed66a47f768b4edd975f420afed071cf (DEFAULT_TABLE table=test-table [id=80ce18ea231b46a8b8a1d48652a03ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.041762 18467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed66a47f768b4edd975f420afed071cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.043391 18684 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Bootstrap starting.
I20260927 03:57:39.043862 18333 tablet_service.cc:1467] Processing CreateTablet for tablet ed66a47f768b4edd975f420afed071cf (DEFAULT_TABLE table=test-table [id=80ce18ea231b46a8b8a1d48652a03ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.044051 18684 tablet_bootstrap.cc:654] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.044186 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed66a47f768b4edd975f420afed071cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.044554 18685 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Bootstrap starting.
I20260927 03:57:39.044586 18684 log.cc:824] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.045219 18685 tablet_bootstrap.cc:654] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.045393 18684 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: No bootstrap required, opened a new log
I20260927 03:57:39.045466 18684 ts_tablet_manager.cc:1397] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:39.045727 18685 log.cc:824] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.046399 18685 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: No bootstrap required, opened a new log
I20260927 03:57:39.046460 18685 ts_tablet_manager.cc:1397] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:39.047227 18688 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Bootstrap starting.
I20260927 03:57:39.047415 18684 raft_consensus.cc:348] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.047591 18684 raft_consensus.cc:374] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.047631 18684 raft_consensus.cc:729] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6c1a5b5b7445e792232a905cc54ca3, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.047755 18684 consensus_queue.cc:260] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.047910 18688 tablet_bootstrap.cc:654] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.048010 18684 ts_tablet_manager.cc:1428] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:57:39.048062 18667 heartbeater.cc:503] Master 127.1.50.126:37919 was elected leader, sending a full tablet report...
I20260927 03:57:39.048481 18688 log.cc:824] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.048666 18685 raft_consensus.cc:348] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.048827 18685 raft_consensus.cc:374] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.048866 18685 raft_consensus.cc:729] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a3bb58f1b443ca9c43d95f21401ca2, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.048974 18685 consensus_queue.cc:260] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.049249 18685 ts_tablet_manager.cc:1428] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:39.049396 18533 heartbeater.cc:503] Master 127.1.50.126:37919 was elected leader, sending a full tablet report...
I20260927 03:57:39.049484 18688 tablet_bootstrap.cc:492] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: No bootstrap required, opened a new log
I20260927 03:57:39.049557 18688 ts_tablet_manager.cc:1397] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:39.051209 18688 raft_consensus.cc:348] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.051375 18688 raft_consensus.cc:374] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.051415 18688 raft_consensus.cc:729] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6c9c24651148f8a9c216b986e0d457, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.051527 18688 consensus_queue.cc:260] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.051762 18688 ts_tablet_manager.cc:1428] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:39.051800 18399 heartbeater.cc:503] Master 127.1.50.126:37919 was elected leader, sending a full tablet report...
W20260927 03:57:39.106303 18534 tablet.cc:2369] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:39.202898 18400 tablet.cc:2369] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:39.231318 18692 raft_consensus.cc:482] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:39.231472 18692 raft_consensus.cc:504] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.231787 18692 leader_election.cc:288] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce6c1a5b5b7445e792232a905cc54ca3 (127.1.50.67:42917), 78a3bb58f1b443ca9c43d95f21401ca2 (127.1.50.66:35291)
I20260927 03:57:39.235888 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.232115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37981 (local address 127.1.50.67:42917)
0927 03:57:39.232173 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.232176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.232243 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:57:39.232338 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.232339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.232353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.232415 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.232583 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.233321 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.234865 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.235479 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.235564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.235577 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.235776 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.235819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:57:39.235934 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.232217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:35291 (local address 127.1.50.65:44537)
0927 03:57:39.232304 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.232313 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.232355 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.232500 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.232501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.232502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.232563 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.232565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.233437 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.233439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.234759 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.234762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.235594 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.235595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.235687 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.235692 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.235756 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.235809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":8,"thread_start_us":46,"threads_started":1}
I20260927 03:57:39.235888 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.232287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:44537 (local address 127.1.50.66:35291)
0927 03:57:39.232331 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.232333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.232345 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:57:39.232436 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.232437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.232448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.232497 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.232627 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.233360 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.234865 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.235505 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.235568 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.235577 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.235768 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.235813 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:57:39.236016 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.232018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42917 (local address 127.1.50.65:37981)
0927 03:57:39.232169 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.232181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.232264 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.232428 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.232428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.232430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.232512 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.232515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.233437 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.233439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.234759 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.234762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.235592 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.235594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.235697 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.235699 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.235756 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.235808 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":111,"thread_start_us":84,"threads_started":1}
I20260927 03:57:39.236330 18487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed66a47f768b4edd975f420afed071cf" candidate_uuid: "0f6c9c24651148f8a9c216b986e0d457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" is_pre_election: true
I20260927 03:57:39.236336 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed66a47f768b4edd975f420afed071cf" candidate_uuid: "0f6c9c24651148f8a9c216b986e0d457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" is_pre_election: true
I20260927 03:57:39.236490 18487 raft_consensus.cc:2454] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6c9c24651148f8a9c216b986e0d457 in term 0.
I20260927 03:57:39.236505 18621 raft_consensus.cc:2454] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6c9c24651148f8a9c216b986e0d457 in term 0.
I20260927 03:57:39.236729 18286 leader_election.cc:302] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6c9c24651148f8a9c216b986e0d457, 78a3bb58f1b443ca9c43d95f21401ca2; no voters:
I20260927 03:57:39.236876 18692 raft_consensus.cc:2781] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:39.236943 18692 raft_consensus.cc:482] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:39.236975 18692 raft_consensus.cc:3037] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.237669 18692 raft_consensus.cc:504] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.237784 18692 leader_election.cc:288] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [CANDIDATE]: Term 1 election: Requested vote from peers ce6c1a5b5b7445e792232a905cc54ca3 (127.1.50.67:42917), 78a3bb58f1b443ca9c43d95f21401ca2 (127.1.50.66:35291)
I20260927 03:57:39.237958 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed66a47f768b4edd975f420afed071cf" candidate_uuid: "0f6c9c24651148f8a9c216b986e0d457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c1a5b5b7445e792232a905cc54ca3"
I20260927 03:57:39.238054 18487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed66a47f768b4edd975f420afed071cf" candidate_uuid: "0f6c9c24651148f8a9c216b986e0d457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a3bb58f1b443ca9c43d95f21401ca2"
I20260927 03:57:39.238062 18621 raft_consensus.cc:3037] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.238137 18487 raft_consensus.cc:3037] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.238797 18621 raft_consensus.cc:2454] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6c9c24651148f8a9c216b986e0d457 in term 1.
I20260927 03:57:39.238801 18487 raft_consensus.cc:2454] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6c9c24651148f8a9c216b986e0d457 in term 1.
I20260927 03:57:39.238955 18286 leader_election.cc:302] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6c9c24651148f8a9c216b986e0d457, ce6c1a5b5b7445e792232a905cc54ca3; no voters:
I20260927 03:57:39.239032 18692 raft_consensus.cc:2781] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:39.239158 18692 raft_consensus.cc:686] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6c9c24651148f8a9c216b986e0d457, State: Running, Role: LEADER
I20260927 03:57:39.239236 18692 consensus_queue.cc:237] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } }
I20260927 03:57:39.240111 18208 catalog_manager.cc:5495] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6c9c24651148f8a9c216b986e0d457 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "0f6c9c24651148f8a9c216b986e0d457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6c9c24651148f8a9c216b986e0d457" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 36037 } health_report { overall_health: HEALTHY } } }
W20260927 03:57:39.256189 18668 tablet.cc:2369] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:39.266680 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.264097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.1.50.65:36037)
0927 03:57:39.264193 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.264196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.264207 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:39.264252 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.264253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.264275 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.264342 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:39.264547 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.265059 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.265828 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.266115 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.266172 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.266376 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:39.266491 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:39.266563 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.266583 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.266604 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:57:39.268415 18621 raft_consensus.cc:1261] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 1 FOLLOWER]: Refusing update from remote peer 0f6c9c24651148f8a9c216b986e0d457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:39.268428 18487 raft_consensus.cc:1261] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 1 FOLLOWER]: Refusing update from remote peer 0f6c9c24651148f8a9c216b986e0d457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:39.268720 18692 consensus_queue.cc:1035] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6c1a5b5b7445e792232a905cc54ca3" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:39.268857 18695 consensus_queue.cc:1035] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a3bb58f1b443ca9c43d95f21401ca2" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:39.271756 18700 mvcc.cc:205] Tried to move back new op lower bound from 7333812057160970240 to 7333812057044545536. Current Snapshot: MvccSnapshot[applied={T|T < 7333812057160970240}]
I20260927 03:57:39.273581 18206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:39.273669 18206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:39.274420 18206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63b6fda396be4a59bb780ec47cc8af84: Sending DeleteTablet for 3 replicas of tablet ed66a47f768b4edd975f420afed071cf
I20260927 03:57:39.274750 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet ed66a47f768b4edd975f420afed071cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:39 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:39798
I20260927 03:57:39.274806 18467 tablet_service.cc:1514] Processing DeleteTablet for tablet ed66a47f768b4edd975f420afed071cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:39 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:54192
I20260927 03:57:39.274825 18333 tablet_service.cc:1514] Processing DeleteTablet for tablet ed66a47f768b4edd975f420afed071cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:39 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:33914
I20260927 03:57:39.275029 18710 tablet_replica.cc:330] T ed66a47f768b4edd975f420afed071cf P 0f6c9c24651148f8a9c216b986e0d457: stopping tablet replica
I20260927 03:57:39.275233 18708 tablet_replica.cc:330] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: stopping tablet replica
I20260927 03:57:39.275331 18708 raft_consensus.cc:2229] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:39.275435 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18270
I20260927 03:57:39.275446 18708 raft_consensus.cc:2258] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:39.276768 18708 ts_tablet_manager.cc:1905] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:39.278671 18708 ts_tablet_manager.cc:1918] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:39.278789 18708 log.cc:1196] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/ed66a47f768b4edd975f420afed071cf
I20260927 03:57:39.278872 18709 tablet_replica.cc:330] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: stopping tablet replica
I20260927 03:57:39.279040 18709 raft_consensus.cc:2229] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:39.279078 18708 ts_tablet_manager.cc:1939] T ed66a47f768b4edd975f420afed071cf P ce6c1a5b5b7445e792232a905cc54ca3: Deleting consensus metadata
I20260927 03:57:39.279227 18709 raft_consensus.cc:2258] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:39.279531 18192 catalog_manager.cc:4841] TS ce6c1a5b5b7445e792232a905cc54ca3 (127.1.50.67:42917): tablet ed66a47f768b4edd975f420afed071cf (table test-table [id=80ce18ea231b46a8b8a1d48652a03ba0]) successfully deleted
I20260927 03:57:39.279680 18709 ts_tablet_manager.cc:1905] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:39.281342 18709 ts_tablet_manager.cc:1918] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:39.281419 18709 log.cc:1196] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/ed66a47f768b4edd975f420afed071cf
I20260927 03:57:39.281687 18709 ts_tablet_manager.cc:1939] T ed66a47f768b4edd975f420afed071cf P 78a3bb58f1b443ca9c43d95f21401ca2: Deleting consensus metadata
I20260927 03:57:39.282279 18192 catalog_manager.cc:4841] TS 78a3bb58f1b443ca9c43d95f21401ca2 (127.1.50.66:35291): tablet ed66a47f768b4edd975f420afed071cf (table test-table [id=80ce18ea231b46a8b8a1d48652a03ba0]) successfully deleted
W20260927 03:57:39.283540 18194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:36037 (error 108)
I20260927 03:57:39.283682 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18404
I20260927 03:57:39.284427 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.284288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:36037 (local address 127.0.0.1:33932)
0927 03:57:39.284347 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.284391 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:36037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:57:39.284577 18194 catalog_manager.cc:4584] TS 0f6c9c24651148f8a9c216b986e0d457 (127.1.50.65:36037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed66a47f768b4edd975f420afed071cf: Network error: Client connection negotiation failed: client connection to 127.1.50.65:36037: connect: Connection refused (error 111)
I20260927 03:57:39.288966 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18538
I20260927 03:57:39.293057 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18176
2026-09-27T03:57:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2011 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T03:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:39Z Disabled control of system clock
I20260927 03:57:39.319125 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39127
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:40063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:39.400403 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:39.400637 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:39.400658 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:39.401863 18715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:39.401906 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:39.401916 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:39.401927 18715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:39.401937 18715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:39.403548 18715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:39.403779 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:39.404042 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:39.408959 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.409006 18715 server_base.cc:1034] running on GCE node
W20260927 03:57:39.408972 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.408987 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.409497 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:39.409777 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:39.410943 18715 hybrid_clock.cc:648] HybridClock initialized: now 1790481459410924 us; error 38 us; skew 500 ppm
I20260927 03:57:39.412333 18715 webserver.cc:460] Webserver started at http://127.1.50.126:44069/ using document root <none> and password file <none>
I20260927 03:57:39.412586 18715 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:39.412638 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:39.412767 18715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:39.413828 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "a75f17b5d8864272b68b98c504759621"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "a9c96691704ca320a80cef27c20bdd76"
server_key_iv: "841066d9793ee8173901298f63075ba9"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.414186 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "a75f17b5d8864272b68b98c504759621"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "a9c96691704ca320a80cef27c20bdd76"
server_key_iv: "841066d9793ee8173901298f63075ba9"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.415416 18715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:39.416181 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.416332 18715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:57:39.416414 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "a75f17b5d8864272b68b98c504759621"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "a9c96691704ca320a80cef27c20bdd76"
server_key_iv: "841066d9793ee8173901298f63075ba9"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.416476 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:39.431469 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:39.431794 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:39.431926 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:39.435716 18715 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:39127
I20260927 03:57:39.435760 18781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:39127 every 8 connection(s)
I20260927 03:57:39.436074 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:39.436985 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.439263 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: Bootstrap starting.
I20260927 03:57:39.439824 18782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.440109 18782 log.cc:824] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.440771 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: No bootstrap required, opened a new log
I20260927 03:57:39.442366 18782 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } }
I20260927 03:57:39.442534 18782 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.442559 18782 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75f17b5d8864272b68b98c504759621, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.442680 18782 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } }
I20260927 03:57:39.442751 18782 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:39.442773 18782 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:39.442798 18782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.443336 18782 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } }
I20260927 03:57:39.443406 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18715
I20260927 03:57:39.443430 18782 leader_election.cc:302] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75f17b5d8864272b68b98c504759621; no voters:
I20260927 03:57:39.443521 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:39.443590 18782 leader_election.cc:288] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:39.443655 18785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:39.443712 1225 external_mini_cluster.cc:1428] Setting key 83e34cbb5a66890a8226c50de821f75c
I20260927 03:57:39.443819 18785 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [term 1 LEADER]: Becoming Leader. State: Replica: a75f17b5d8864272b68b98c504759621, State: Running, Role: LEADER
I20260927 03:57:39.443866 18782 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:39.443996 18785 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } }
I20260927 03:57:39.444428 18786 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75f17b5d8864272b68b98c504759621. Latest consensus state: current_term: 1 leader_uuid: "a75f17b5d8864272b68b98c504759621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } } }
I20260927 03:57:39.444509 18787 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75f17b5d8864272b68b98c504759621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75f17b5d8864272b68b98c504759621" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39127 } } }
I20260927 03:57:39.444617 18786 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:39.444958 18787 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:39.445150 18796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:39.445637 18796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:39.447154 18796 catalog_manager.cc:1347] Generated new cluster ID: faa657ab5f0f4dee815243863405e861
I20260927 03:57:39.447199 18796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:39.449820 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.444274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.1.50.126:39127)
0927 03:57:39.444748 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.444757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.444789 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:57:39.444905 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.444913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.444990 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.445419 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.445534 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.446681 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.447751 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.448357 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.448474 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.449148 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.449152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.449153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:39.449352 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.449388 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:39.449402 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.449462 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.449543 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.449617 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":66,"threads_started":1}
I20260927 03:57:39.457634 18796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:39.458282 18796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:39.464802 18796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: Generated new TSK 0
I20260927 03:57:39.464993 18796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:39.466933 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--builtin_ntp_servers=127.1.50.84:40063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:39.547823 18806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:39.548048 18806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:39.548071 18806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:39.549279 18806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:39.549322 18806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:39.549348 18806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:39.550797 18806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:39.551047 18806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:39.551347 18806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:57:39.557830 18806 server_base.cc:1034] running on GCE node
W20260927 03:57:39.557690 18814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.557703 18811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.557741 18812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.558194 18806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:39.558444 18806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:39.559607 18806 hybrid_clock.cc:648] HybridClock initialized: now 1790481459559588 us; error 34 us; skew 500 ppm
I20260927 03:57:39.560875 18806 webserver.cc:460] Webserver started at http://127.1.50.65:37237/ using document root <none> and password file <none>
I20260927 03:57:39.561182 18806 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:39.561236 18806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:39.561362 18806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:39.562479 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "32c094e09c0b455bbc38d244fcd7d951"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "f85ff0cd9896c5263cad6bfe8a2766cc"
server_key_iv: "6845ef6783b72e9a9e16713a0f7e0320"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.562810 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "32c094e09c0b455bbc38d244fcd7d951"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "f85ff0cd9896c5263cad6bfe8a2766cc"
server_key_iv: "6845ef6783b72e9a9e16713a0f7e0320"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.564069 18806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:39.564776 18820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.564960 18806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:39.565062 18806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "32c094e09c0b455bbc38d244fcd7d951"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "f85ff0cd9896c5263cad6bfe8a2766cc"
server_key_iv: "6845ef6783b72e9a9e16713a0f7e0320"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.565136 18806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:39.581068 18806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:39.581377 18806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:39.581487 18806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:39.581750 18806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:39.582110 18806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:39.582146 18806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.582170 18806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:39.582180 18806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.588058 18806 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:34225
I20260927 03:57:39.588133 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:34225 every 8 connection(s)
I20260927 03:57:39.588435 18806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:39.591972 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18806
I20260927 03:57:39.592074 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:39.592260 1225 external_mini_cluster.cc:1428] Setting key d275dae7b2bcef0c168741d4a00d4ce6
I20260927 03:57:39.593367 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--builtin_ntp_servers=127.1.50.84:40063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:57:39.594434 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.589188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:32925 (local address 127.1.50.126:39127)
0927 03:57:39.589268 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.589270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.589522 (+ 252us) server_negotiation.cc:407] Connection header received
0927 03:57:39.590089 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.590091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.590107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.590172 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.590994 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.591561 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.592832 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.593122 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.593189 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.594175 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.594176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.594176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:39.594181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.594197 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:39.594201 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.594242 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.594320 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.594354 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:57:39.594652 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.589190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39127 (local address 127.1.50.65:32925)
0927 03:57:39.589368 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.589410 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.589912 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.590343 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.590347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.590361 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:39.590875 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.590879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.591698 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.591700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.592736 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.592740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.593220 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.593221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.593369 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.593937 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:39.593938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:39.594012 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:39.594089 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:39.594249 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:39.594254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:39.594257 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.594320 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.594427 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260927 03:57:39.595142 18934 heartbeater.cc:347] Connected to a master server at 127.1.50.126:39127
I20260927 03:57:39.595252 18934 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:39.595461 18934 heartbeater.cc:511] Master 127.1.50.126:39127 requested a full tablet report, sending...
I20260927 03:57:39.596045 18746 ts_manager.cc:194] Registered new tserver with Master: 32c094e09c0b455bbc38d244fcd7d951 (127.1.50.65:34225)
I20260927 03:57:39.596900 18746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.65:32925
W20260927 03:57:39.675038 18937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:39.675282 18937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:39.675305 18937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:39.676558 18937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:39.676608 18937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:39.676636 18937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:39.678251 18937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:39.678522 18937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:39.678853 18937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:39.684141 18943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.684158 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.684217 18937 server_base.cc:1034] running on GCE node
W20260927 03:57:39.684136 18942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.684497 18937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:39.684729 18937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:39.685881 18937 hybrid_clock.cc:648] HybridClock initialized: now 1790481459685866 us; error 30 us; skew 500 ppm
I20260927 03:57:39.687178 18937 webserver.cc:460] Webserver started at http://127.1.50.66:38639/ using document root <none> and password file <none>
I20260927 03:57:39.687413 18937 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:39.687462 18937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:39.687595 18937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:39.688979 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "23fbee02f1c74789a9ac2adfe6809a5e"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "4ad3fb12bcf0cb5a95398b834a5d029f"
server_key_iv: "63adb18fe5fddf1fb29be2b0d51aa508"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.689499 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "23fbee02f1c74789a9ac2adfe6809a5e"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "4ad3fb12bcf0cb5a95398b834a5d029f"
server_key_iv: "63adb18fe5fddf1fb29be2b0d51aa508"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.691136 18937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260927 03:57:39.692005 18951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.692281 18937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:39.692374 18937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "23fbee02f1c74789a9ac2adfe6809a5e"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "4ad3fb12bcf0cb5a95398b834a5d029f"
server_key_iv: "63adb18fe5fddf1fb29be2b0d51aa508"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.692446 18937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:39.701937 18937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:39.702301 18937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:39.702435 18937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:39.702659 18937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:39.702965 18937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:39.702998 18937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.703027 18937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:39.703044 18937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.708480 18937 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:34343
I20260927 03:57:39.708520 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:34343 every 8 connection(s)
I20260927 03:57:39.708848 18937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:57:39.713912 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.709567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:40563 (local address 127.1.50.126:39127)
0927 03:57:39.709619 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.709621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.709832 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:57:39.710393 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.710395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.710406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.710458 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.711115 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.711612 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.712558 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.712745 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.712797 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.713644 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.713645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.713645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:39.713650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.713663 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:39.713667 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.713708 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.713792 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.713820 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:57:39.714169 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.709500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39127 (local address 127.1.50.66:40563)
0927 03:57:39.709718 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.709751 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.710290 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.710540 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.710544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.710554 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:39.711029 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.711033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.711691 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.711692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.712477 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.712479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.712813 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.712814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.712942 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.713484 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:39.713485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:39.713514 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:39.713546 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:39.713718 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:39.713723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:39.713726 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.713788 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.713895 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260927 03:57:39.714600 19065 heartbeater.cc:347] Connected to a master server at 127.1.50.126:39127
I20260927 03:57:39.714684 19065 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:39.714823 19065 heartbeater.cc:511] Master 127.1.50.126:39127 requested a full tablet report, sending...
I20260927 03:57:39.715220 18746 ts_manager.cc:194] Registered new tserver with Master: 23fbee02f1c74789a9ac2adfe6809a5e (127.1.50.66:34343)
I20260927 03:57:39.715571 18746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.66:40563
I20260927 03:57:39.718909 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 18937
I20260927 03:57:39.718993 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:39.719177 1225 external_mini_cluster.cc:1428] Setting key 60f9d13896dae170bf13a1a9607728b5
I20260927 03:57:39.720160 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--builtin_ntp_servers=127.1.50.84:40063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:39.799337 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:39.799582 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:39.799614 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:39.800882 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:39.800935 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:39.800979 19068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:39.802419 19068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:39127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:39.802654 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:39.802933 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:39.807951 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.807960 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:39.807961 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:39.808166 19068 server_base.cc:1034] running on GCE node
I20260927 03:57:39.808493 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:39.808738 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:39.809880 19068 hybrid_clock.cc:648] HybridClock initialized: now 1790481459809867 us; error 41 us; skew 500 ppm
I20260927 03:57:39.811254 19068 webserver.cc:460] Webserver started at http://127.1.50.67:33407/ using document root <none> and password file <none>
I20260927 03:57:39.811486 19068 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:39.811535 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:39.811658 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:39.812701 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "8fdcebafb087432ab035033e052dc562"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "16033d1bdc42e1780cdf5f7793280a9d"
server_key_iv: "05eb52c0ffa140719b8399c55b4bf074"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.813040 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "8fdcebafb087432ab035033e052dc562"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "16033d1bdc42e1780cdf5f7793280a9d"
server_key_iv: "05eb52c0ffa140719b8399c55b4bf074"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.814311 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:57:39.815084 19082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.815235 19068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:39.815315 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "8fdcebafb087432ab035033e052dc562"
format_stamp: "Formatted at 2026-09-27 03:57:39 on dist-test-slave-96fg"
server_key: "16033d1bdc42e1780cdf5f7793280a9d"
server_key_iv: "05eb52c0ffa140719b8399c55b4bf074"
server_key_version: "encryptionkey@0"
I20260927 03:57:39.815380 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:39.835951 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:39.836266 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:39.836395 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:39.836625 19068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:39.836948 19068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:39.836982 19068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.837011 19068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:39.837035 19068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:39.842599 19068 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:35251
I20260927 03:57:39.842639 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:35251 every 8 connection(s)
I20260927 03:57:39.842960 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:39.845037 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19068
I20260927 03:57:39.845162 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:39.845376 1225 external_mini_cluster.cc:1428] Setting key 3c291731f668cb5226f5755db90220b7
I20260927 03:57:39.848714 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.843716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:37619 (local address 127.1.50.126:39127)
0927 03:57:39.843806 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.843808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.844057 (+ 249us) server_negotiation.cc:407] Connection header received
0927 03:57:39.844643 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.844644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.844657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.844729 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.845704 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.846403 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.847412 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.847613 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.847672 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.848511 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.848512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.848513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:39.848517 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.848530 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:39.848534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.848573 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.848628 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.848653 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 03:57:39.848907 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.843678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39127 (local address 127.1.50.67:37619)
0927 03:57:39.843847 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.843889 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.844472 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.844837 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.844841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.844852 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:39.845631 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.845636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.846469 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.846469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.847315 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.847317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.847697 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.847698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.847812 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.848360 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:39.848360 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:39.848390 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:39.848415 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:39.848576 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:39.848578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:39.848581 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.848623 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.848706 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260927 03:57:39.849345 19196 heartbeater.cc:347] Connected to a master server at 127.1.50.126:39127
I20260927 03:57:39.849435 19196 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:39.849611 19196 heartbeater.cc:511] Master 127.1.50.126:39127 requested a full tablet report, sending...
I20260927 03:57:39.849927 18746 ts_manager.cc:194] Registered new tserver with Master: 8fdcebafb087432ab035033e052dc562 (127.1.50.67:35251)
I20260927 03:57:39.850385 18746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.67:37619
I20260927 03:57:39.857457 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:39.861948 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.858844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.1.50.126:39127)
0927 03:57:39.858907 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.858909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.859010 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:57:39.859060 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.859061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.859072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.859131 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:39.859357 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.859865 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.860670 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.860859 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.860923 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.861653 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:39.861654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:39.861654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:39.861659 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:39.861672 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:39.861676 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:39.861728 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.861857 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.861883 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:57:39.863706 18746 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:39.865265 18746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:39.872443 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:34343 (local address 127.0.0.1:37642)
0927 03:57:39.867821 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.867865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.867941 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.868410 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.868410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.868412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.868460 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.868462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.869763 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.869763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.871176 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.871179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.872080 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.872166 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872180 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.872268 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.872328 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":28,"thread_start_us":70,"threads_started":1}
I20260927 03:57:39.872736 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.1.50.66:34343)
0927 03:57:39.867837 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.867843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.867924 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:57:39.868102 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.868105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.868190 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.868340 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.868518 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.869696 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.871261 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.871986 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872050 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872125 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.872293 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.872333 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260927 03:57:39.872776 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.1.50.65:34225)
0927 03:57:39.867991 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.867995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.868084 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:57:39.868162 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.868164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.868183 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.868255 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.869786 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.870645 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.871627 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.872220 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872279 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.872649 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.872689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260927 03:57:39.872593 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34225 (local address 127.0.0.1:40730)
0927 03:57:39.868002 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.868012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.868088 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.868344 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.868346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.868348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.868460 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.868462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.870715 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.870716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.871514 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.871516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.872390 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.872445 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872448 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.872495 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.872529 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":65,"threads_started":1}
I20260927 03:57:39.873132 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:35251 (local address 127.0.0.1:38050)
0927 03:57:39.868158 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.868168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.868213 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.868493 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.868493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.868495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.868527 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.868528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.869739 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.869740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.871176 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.871179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.872908 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.872983 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872987 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.873035 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.873069 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":72,"thread_start_us":55,"threads_started":1}
I20260927 03:57:39.873358 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.867982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.1.50.67:35251)
0927 03:57:39.868315 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.868319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.868327 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:39.868362 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.868364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.868377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.868442 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.868584 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.869662 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872157 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.872835 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.872906 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.872982 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.873232 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.873274 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":88,"threads_started":1}
I20260927 03:57:39.873872 18999 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5e760474840499c22cba2f1664c34 (DEFAULT_TABLE table=test-table [id=b5d2cacbb5614bbe86a5716d15644ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.873872 18868 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5e760474840499c22cba2f1664c34 (DEFAULT_TABLE table=test-table [id=b5d2cacbb5614bbe86a5716d15644ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.873919 19130 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5e760474840499c22cba2f1664c34 (DEFAULT_TABLE table=test-table [id=b5d2cacbb5614bbe86a5716d15644ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:39.874259 19130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b5e760474840499c22cba2f1664c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.874262 18868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b5e760474840499c22cba2f1664c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.874262 18999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b5e760474840499c22cba2f1664c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:39.877022 19210 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Bootstrap starting.
I20260927 03:57:39.877391 19211 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: Bootstrap starting.
I20260927 03:57:39.877576 19212 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Bootstrap starting.
I20260927 03:57:39.877719 19210 tablet_bootstrap.cc:654] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.878063 19211 tablet_bootstrap.cc:654] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.878134 19210 log.cc:824] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.878312 19212 tablet_bootstrap.cc:654] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:39.878402 19211 log.cc:824] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.878692 19212 log.cc:824] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:39.879182 19210 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: No bootstrap required, opened a new log
I20260927 03:57:39.879240 19210 ts_tablet_manager.cc:1397] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:39.879355 19212 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: No bootstrap required, opened a new log
I20260927 03:57:39.879424 19212 ts_tablet_manager.cc:1397] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:39.881080 19210 raft_consensus.cc:348] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.881222 19210 raft_consensus.cc:374] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.881242 19210 raft_consensus.cc:729] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcebafb087432ab035033e052dc562, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.881244 19212 raft_consensus.cc:348] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.881345 19210 consensus_queue.cc:260] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.881400 19212 raft_consensus.cc:374] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.881451 19212 raft_consensus.cc:729] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fbee02f1c74789a9ac2adfe6809a5e, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.881558 19212 consensus_queue.cc:260] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.881599 19210 ts_tablet_manager.cc:1428] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:57:39.881675 19196 heartbeater.cc:503] Master 127.1.50.126:39127 was elected leader, sending a full tablet report...
I20260927 03:57:39.881814 19212 ts_tablet_manager.cc:1428] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:57:39.881860 19065 heartbeater.cc:503] Master 127.1.50.126:39127 was elected leader, sending a full tablet report...
I20260927 03:57:39.881986 19211 tablet_bootstrap.cc:492] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: No bootstrap required, opened a new log
I20260927 03:57:39.882102 19211 ts_tablet_manager.cc:1397] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:57:39.884117 19211 raft_consensus.cc:348] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.884250 19211 raft_consensus.cc:374] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:39.884270 19211 raft_consensus.cc:729] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c094e09c0b455bbc38d244fcd7d951, State: Initialized, Role: FOLLOWER
I20260927 03:57:39.884346 19211 consensus_queue.cc:260] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.884531 19211 ts_tablet_manager.cc:1428] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 03:57:39.884586 18934 heartbeater.cc:503] Master 127.1.50.126:39127 was elected leader, sending a full tablet report...
W20260927 03:57:39.959378 19066 tablet.cc:2369] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:39.984104 19217 raft_consensus.cc:482] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:39.984264 19217 raft_consensus.cc:504] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.984645 19217 leader_election.cc:288] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32c094e09c0b455bbc38d244fcd7d951 (127.1.50.65:34225), 8fdcebafb087432ab035033e052dc562 (127.1.50.67:35251)
I20260927 03:57:39.987329 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.984890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:43657 (local address 127.1.50.65:34225)
0927 03:57:39.984939 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.984941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.984950 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:39.985011 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.985012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.985023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.985068 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.985240 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.985745 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.986682 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.987044 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.987101 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.987111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.987222 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.987257 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:57:39.987329 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.984830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34225 (local address 127.1.50.66:43657)
0927 03:57:39.984881 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.984893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.984944 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.985075 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.985076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.985077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.985146 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.985147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.985821 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.985822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.986604 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.986606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.987108 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.987109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.987166 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.987169 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.987212 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.987256 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:57:39.987488 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.985159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:35207 (local address 127.1.50.67:35251)
0927 03:57:39.985220 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:39.985222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:39.985229 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:57:39.985256 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:39.985257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:39.985264 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:39.985313 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:39.985440 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.985894 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.986822 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.987179 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.987238 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.987247 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:39.987382 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:57:39.987421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:57:39.987488 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:39.984946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:35251 (local address 127.1.50.66:35207)
0927 03:57:39.985114 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:39.985123 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:39.985178 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:39.985320 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:39.985321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:39.985322 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:39.985388 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.985389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.985966 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.985967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.986731 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:39.986733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:39.987246 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:39.987247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:39.987324 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:39.987328 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:39.987371 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:57:39.987407 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260927 03:57:39.987656 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5e760474840499c22cba2f1664c34" candidate_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c094e09c0b455bbc38d244fcd7d951" is_pre_election: true
I20260927 03:57:39.987812 18888 raft_consensus.cc:2454] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23fbee02f1c74789a9ac2adfe6809a5e in term 0.
I20260927 03:57:39.988054 18952 leader_election.cc:302] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fbee02f1c74789a9ac2adfe6809a5e, 32c094e09c0b455bbc38d244fcd7d951; no voters:
I20260927 03:57:39.988184 19217 raft_consensus.cc:2781] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:39.988198 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5e760474840499c22cba2f1664c34" candidate_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcebafb087432ab035033e052dc562" is_pre_election: true
I20260927 03:57:39.988265 19217 raft_consensus.cc:482] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:39.988301 19217 raft_consensus.cc:3037] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.988330 19150 raft_consensus.cc:2454] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23fbee02f1c74789a9ac2adfe6809a5e in term 0.
I20260927 03:57:39.989042 19217 raft_consensus.cc:504] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.989147 19217 leader_election.cc:288] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [CANDIDATE]: Term 1 election: Requested vote from peers 32c094e09c0b455bbc38d244fcd7d951 (127.1.50.65:34225), 8fdcebafb087432ab035033e052dc562 (127.1.50.67:35251)
I20260927 03:57:39.989324 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5e760474840499c22cba2f1664c34" candidate_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c094e09c0b455bbc38d244fcd7d951"
I20260927 03:57:39.989351 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5e760474840499c22cba2f1664c34" candidate_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcebafb087432ab035033e052dc562"
I20260927 03:57:39.989406 19150 raft_consensus.cc:3037] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.989411 18888 raft_consensus.cc:3037] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:39.990160 19150 raft_consensus.cc:2454] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23fbee02f1c74789a9ac2adfe6809a5e in term 1.
I20260927 03:57:39.990242 18888 raft_consensus.cc:2454] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23fbee02f1c74789a9ac2adfe6809a5e in term 1.
I20260927 03:57:39.990309 18952 leader_election.cc:302] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fbee02f1c74789a9ac2adfe6809a5e, 8fdcebafb087432ab035033e052dc562; no voters:
I20260927 03:57:39.990402 19217 raft_consensus.cc:2781] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:39.990562 19217 raft_consensus.cc:686] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 1 LEADER]: Becoming Leader. State: Replica: 23fbee02f1c74789a9ac2adfe6809a5e, State: Running, Role: LEADER
I20260927 03:57:39.990660 19217 consensus_queue.cc:237] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } }
I20260927 03:57:39.991449 18746 catalog_manager.cc:5495] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 23fbee02f1c74789a9ac2adfe6809a5e (127.1.50.66). New cstate: current_term: 1 leader_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fbee02f1c74789a9ac2adfe6809a5e" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 34343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 } health_report { overall_health: UNKNOWN } } }
I20260927 03:57:40.017614 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:40.015126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.1.50.66:34343)
0927 03:57:40.015167 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:40.015169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:40.015177 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:40.015253 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:40.015254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:40.015270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:40.015320 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:40.015470 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:40.015932 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:40.016691 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:40.016935 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:40.016997 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:40.017176 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:40.017305 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:40.017388 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:40.017473 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:57:40.017499 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:57:40.020473 19150 raft_consensus.cc:1261] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 1 FOLLOWER]: Refusing update from remote peer 23fbee02f1c74789a9ac2adfe6809a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:40.020483 18888 raft_consensus.cc:1261] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 1 FOLLOWER]: Refusing update from remote peer 23fbee02f1c74789a9ac2adfe6809a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:40.020792 19217 consensus_queue.cc:1035] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcebafb087432ab035033e052dc562" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:40.020943 19220 consensus_queue.cc:1035] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32c094e09c0b455bbc38d244fcd7d951" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:40.023211 19223 mvcc.cc:205] Tried to move back new op lower bound from 7333812060241113088 to 7333812060122288128. Current Snapshot: MvccSnapshot[applied={T|T < 7333812060241113088}]
I20260927 03:57:40.023794 19225 mvcc.cc:205] Tried to move back new op lower bound from 7333812060241113088 to 7333812060122288128. Current Snapshot: MvccSnapshot[applied={T|T < 7333812060241113088}]
I20260927 03:57:40.025918 18744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:40.026095 18744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:40.026793 18744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a75f17b5d8864272b68b98c504759621: Sending DeleteTablet for 3 replicas of tablet 22b5e760474840499c22cba2f1664c34
I20260927 03:57:40.027189 18868 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5e760474840499c22cba2f1664c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:40 UTC) from {username='slave'} at 127.0.0.1:40730
I20260927 03:57:40.027237 18999 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5e760474840499c22cba2f1664c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:40 UTC) from {username='slave'} at 127.0.0.1:37642
I20260927 03:57:40.027428 19233 tablet_replica.cc:330] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951: stopping tablet replica
I20260927 03:57:40.027413 19130 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5e760474840499c22cba2f1664c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:40 UTC) from {username='slave'} at 127.0.0.1:38050
I20260927 03:57:40.027529 19233 raft_consensus.cc:2229] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:40.027602 19234 tablet_replica.cc:330] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: stopping tablet replica
I20260927 03:57:40.027629 19233 raft_consensus.cc:2258] T 22b5e760474840499c22cba2f1664c34 P 32c094e09c0b455bbc38d244fcd7d951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:40.027655 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18806
I20260927 03:57:40.027688 19234 raft_consensus.cc:2229] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:57:40.028488 19234 raft_consensus.cc:2258] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:40.028927 19234 ts_tablet_manager.cc:1905] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:40.031060 19234 ts_tablet_manager.cc:1918] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:40.031139 19234 log.cc:1196] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/22b5e760474840499c22cba2f1664c34
I20260927 03:57:40.031383 19234 ts_tablet_manager.cc:1939] T 22b5e760474840499c22cba2f1664c34 P 23fbee02f1c74789a9ac2adfe6809a5e: Deleting consensus metadata
I20260927 03:57:40.031776 18730 catalog_manager.cc:4841] TS 23fbee02f1c74789a9ac2adfe6809a5e (127.1.50.66:34343): tablet 22b5e760474840499c22cba2f1664c34 (table test-table [id=b5d2cacbb5614bbe86a5716d15644ad6]) successfully deleted
I20260927 03:57:40.034164 19235 tablet_replica.cc:330] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: stopping tablet replica
I20260927 03:57:40.034294 19235 raft_consensus.cc:2229] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:40.034425 19235 raft_consensus.cc:2258] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:57:40.034574 18730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:34225 (error 108)
I20260927 03:57:40.034782 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18937
I20260927 03:57:40.035501 19235 ts_tablet_manager.cc:1905] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:40.036593 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:40.036443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34225 (local address 127.0.0.1:40732)
0927 03:57:40.036503 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:40.036553 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:57:40.036772 18730 catalog_manager.cc:4584] TS 32c094e09c0b455bbc38d244fcd7d951 (127.1.50.65:34225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22b5e760474840499c22cba2f1664c34: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34225: connect: Connection refused (error 111)
I20260927 03:57:40.037057 19235 ts_tablet_manager.cc:1918] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:40.037119 19235 log.cc:1196] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/22b5e760474840499c22cba2f1664c34
I20260927 03:57:40.037324 19235 ts_tablet_manager.cc:1939] T 22b5e760474840499c22cba2f1664c34 P 8fdcebafb087432ab035033e052dc562: Deleting consensus metadata
I20260927 03:57:40.037724 18730 catalog_manager.cc:4841] TS 8fdcebafb087432ab035033e052dc562 (127.1.50.67:35251): tablet 22b5e760474840499c22cba2f1664c34 (table test-table [id=b5d2cacbb5614bbe86a5716d15644ad6]) successfully deleted
I20260927 03:57:40.039736 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19068
I20260927 03:57:40.044348 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 18715
2026-09-27T03:57:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:40 dist-test-slave-96fg krb5kdc[19239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:40 dist-test-slave-96fg krb5kdc[19239](info): set up 2 sockets
Sep 27 03:57:40 dist-test-slave-96fg krb5kdc[19239](info): commencing operation
krb5kdc: starting...
W20260927 03:57:41.017632 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:57:41.143198 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40981
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:41.224148 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:41.224361 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:41.224383 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:41.225615 19252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:41.225672 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:41.225684 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:41.225695 19252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:41.225706 19252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:41.227300 19252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:41.227579 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:41.227835 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:41.233223 19258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.233223 19260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.233228 19257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:41.233259 19252 server_base.cc:1034] running on GCE node
I20260927 03:57:41.233719 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:41.233980 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:41.235167 19252 hybrid_clock.cc:648] HybridClock initialized: now 1790481461235156 us; error 35 us; skew 500 ppm
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:41.237779 19252 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:57:41.238272 19252 webserver.cc:460] Webserver started at http://127.1.50.126:35105/ using document root <none> and password file <none>
I20260927 03:57:41.238498 19252 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:41.238547 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:41.238672 19252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:41.239760 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "b220c65f151541a8968c17aca2c0467d"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "dddc9483c2b089725484f0e08142f5a4"
server_key_iv: "1c682061a996f067eb720d845b634882"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.240104 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "b220c65f151541a8968c17aca2c0467d"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "dddc9483c2b089725484f0e08142f5a4"
server_key_iv: "1c682061a996f067eb720d845b634882"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.241473 19252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:41.242316 19267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.242493 19252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:41.242578 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "b220c65f151541a8968c17aca2c0467d"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "dddc9483c2b089725484f0e08142f5a4"
server_key_iv: "1c682061a996f067eb720d845b634882"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.242643 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:41.258836 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:41.261075 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:41.261220 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:41.265086 19252 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:40981
I20260927 03:57:41.265144 19319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:40981 every 8 connection(s)
I20260927 03:57:41.265460 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:41.266396 19320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:41.268208 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19252
I20260927 03:57:41.268313 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:41.268486 1225 external_mini_cluster.cc:1428] Setting key f7f6bea9e89aa3587eaedacaab68df8e
I20260927 03:57:41.268597 19320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: Bootstrap starting.
I20260927 03:57:41.269241 19320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:41.269506 19320 log.cc:824] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:41.270196 19320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: No bootstrap required, opened a new log
I20260927 03:57:41.272333 19320 raft_consensus.cc:348] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } }
I20260927 03:57:41.272473 19320 raft_consensus.cc:374] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:41.272504 19320 raft_consensus.cc:729] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b220c65f151541a8968c17aca2c0467d, State: Initialized, Role: FOLLOWER
I20260927 03:57:41.272629 19320 consensus_queue.cc:260] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } }
I20260927 03:57:41.272706 19320 raft_consensus.cc:388] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:41.272737 19320 raft_consensus.cc:482] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:41.272781 19320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:41.273326 19320 raft_consensus.cc:504] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } }
I20260927 03:57:41.273420 19320 leader_election.cc:302] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b220c65f151541a8968c17aca2c0467d; no voters:
I20260927 03:57:41.273591 19320 leader_election.cc:288] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:41.273622 19325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:41.273808 19325 raft_consensus.cc:686] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [term 1 LEADER]: Becoming Leader. State: Replica: b220c65f151541a8968c17aca2c0467d, State: Running, Role: LEADER
I20260927 03:57:41.273842 19320 sys_catalog.cc:564] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:41.274009 19325 consensus_queue.cc:237] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } }
I20260927 03:57:41.274487 19327 sys_catalog.cc:455] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b220c65f151541a8968c17aca2c0467d. Latest consensus state: current_term: 1 leader_uuid: "b220c65f151541a8968c17aca2c0467d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } } }
I20260927 03:57:41.274570 19327 sys_catalog.cc:458] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:41.274782 19326 sys_catalog.cc:455] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b220c65f151541a8968c17aca2c0467d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b220c65f151541a8968c17aca2c0467d" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40981 } } }
I20260927 03:57:41.274895 19326 sys_catalog.cc:458] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:41.275070 19334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:41.275517 19334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:41.277110 19334 catalog_manager.cc:1347] Generated new cluster ID: 3c357a47748948b9a7551003649f20f1
I20260927 03:57:41.277163 19334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:41.279464 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.269002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.1.50.126:40981)
0927 03:57:41.269160 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.269167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.269190 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:57:41.269678 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.269685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.269731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.269859 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:41.270622 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.271911 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.272662 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.273155 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.273253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.275956 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:41.275960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:41.275961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:41.275974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:41.277918 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.278326 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.278329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.278329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.278424 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.278622 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.278622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.278623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.278890 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:41.279043 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.279145 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.279270 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260927 03:57:41.287863 19334 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:41.288388 19334 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:41.297232 19334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: Generated new TSK 0
I20260927 03:57:41.297437 19334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:57:41.330221 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:41.428875 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:41.429129 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:41.429165 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:41.430425 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:41.430475 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:41.430514 19346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:41.432036 19346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:41.432286 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:41.432504 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:41.437598 19352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:41.437668 19346 server_base.cc:1034] running on GCE node
W20260927 03:57:41.437674 19351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.437749 19354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:41.437966 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:41.438244 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:41.439435 19346 hybrid_clock.cc:648] HybridClock initialized: now 1790481461439408 us; error 36 us; skew 500 ppm
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:41.442103 19346 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:57:41.442585 19346 webserver.cc:460] Webserver started at http://127.1.50.65:34379/ using document root <none> and password file <none>
I20260927 03:57:41.442785 19346 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:41.442831 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:41.442961 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:41.444005 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "d608b7ba1f7c4b3199b501b19ad23baf"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1171ca1785a3c69c2744f9b341f4afc0"
server_key_iv: "c0bc5b75edd203fb33a9495bbcc50fe2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.444356 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "d608b7ba1f7c4b3199b501b19ad23baf"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1171ca1785a3c69c2744f9b341f4afc0"
server_key_iv: "c0bc5b75edd203fb33a9495bbcc50fe2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.445762 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:57:41.446868 19361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.447124 19346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:41.447218 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "d608b7ba1f7c4b3199b501b19ad23baf"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1171ca1785a3c69c2744f9b341f4afc0"
server_key_iv: "c0bc5b75edd203fb33a9495bbcc50fe2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.447292 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:41.463785 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:41.466253 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:41.466449 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:41.466727 19346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:41.467118 19346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:41.467165 19346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.467206 19346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:41.467229 19346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.472877 19346 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:40121
I20260927 03:57:41.472918 19474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:40121 every 8 connection(s)
I20260927 03:57:41.473234 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:41.476029 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19346
I20260927 03:57:41.476121 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:41.476298 1225 external_mini_cluster.cc:1428] Setting key 3b5be03daf89ecb60d6ed3996bde85ea
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:41.483704 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.473882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:51105 (local address 127.1.50.126:40981)
0927 03:57:41.473926 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.473928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.474174 (+ 246us) server_negotiation.cc:407] Connection header received
0927 03:57:41.474709 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.474710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.474720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.474770 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:41.475680 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.476250 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.477486 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.477699 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.477768 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.480898 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:41.480899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:41.480899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:41.480906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:41.482258 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.482703 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.482703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.482704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.482756 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.483052 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.483054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.483054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.483178 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:41.483240 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.483562 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.483637 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
I20260927 03:57:41.483974 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.473861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40981 (local address 127.1.50.65:51105)
0927 03:57:41.474049 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.474095 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.474632 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.474846 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.474851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.475407 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:41.475600 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.475605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.476332 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.476334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.477389 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.477392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.477781 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.477781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.477919 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.479359 (+ 1440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:41.479367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:41.480790 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:41.482333 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.482341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.482343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.482625 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.482819 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.482820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.482820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.482965 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.483249 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:41.483255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:41.483422 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.483539 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.483698 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260927 03:57:41.484473 19475 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40981
I20260927 03:57:41.484555 19475 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:41.484719 19475 heartbeater.cc:511] Master 127.1.50.126:40981 requested a full tablet report, sending...
I20260927 03:57:41.485301 19284 ts_manager.cc:194] Registered new tserver with Master: d608b7ba1f7c4b3199b501b19ad23baf (127.1.50.65:40121)
I20260927 03:57:41.486006 19284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:51105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:57:41.502128 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:41.581372 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:41.581595 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:41.581629 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:41.582875 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:41.582921 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:41.582957 19480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:41.584470 19480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:41.584710 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:41.584921 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:41.589781 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.589773 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.589776 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:41.590062 19480 server_base.cc:1034] running on GCE node
I20260927 03:57:41.590252 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:41.590485 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:41.591642 19480 hybrid_clock.cc:648] HybridClock initialized: now 1790481461591629 us; error 38 us; skew 500 ppm
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:41.594215 19480 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:57:41.594691 19480 webserver.cc:460] Webserver started at http://127.1.50.66:39479/ using document root <none> and password file <none>
I20260927 03:57:41.594888 19480 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:41.594935 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:41.595058 19480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:41.596055 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "419bef7c1dbc468c803f29a89a295646"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1423ebe151f206f0a992c94eaadea142"
server_key_iv: "f6d1976fb7639aec84c4e18cbee7d2a2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.596386 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "419bef7c1dbc468c803f29a89a295646"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1423ebe151f206f0a992c94eaadea142"
server_key_iv: "f6d1976fb7639aec84c4e18cbee7d2a2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.597623 19480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:41.598328 19495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.598469 19480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:41.598547 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "419bef7c1dbc468c803f29a89a295646"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "1423ebe151f206f0a992c94eaadea142"
server_key_iv: "f6d1976fb7639aec84c4e18cbee7d2a2"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.598608 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:41.610718 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:41.612905 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:41.613049 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:41.613276 19480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:41.613579 19480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:41.613612 19480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.613641 19480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:41.613658 19480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.619464 19480 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:38925
I20260927 03:57:41.619529 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:38925 every 8 connection(s)
I20260927 03:57:41.619892 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:41.627056 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19480
I20260927 03:57:41.627169 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:41.627349 1225 external_mini_cluster.cc:1428] Setting key 3e09c1cb7bd82cda83b8e36480f48b68
I20260927 03:57:41.629734 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.620666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:52399 (local address 127.1.50.126:40981)
0927 03:57:41.620751 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.620752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.620926 (+ 174us) server_negotiation.cc:407] Connection header received
0927 03:57:41.621403 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.621404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.621415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.621474 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:41.622462 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.622955 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.624068 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.624267 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.624331 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.626583 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:41.626584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:41.626584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:41.626591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:41.627974 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.628470 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.628471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.628471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.628530 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.628827 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.628829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.628829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.628954 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:41.629150 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.629531 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.629633 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:57:41.630004 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.620613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40981 (local address 127.1.50.66:52399)
0927 03:57:41.620775 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.620817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.621291 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.621534 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.621537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.622174 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:41.622365 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.622370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.623086 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.623088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.623936 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.623939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.624365 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.624366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.624476 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.624971 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:41.624980 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:41.626469 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:41.628045 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.628053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.628055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.628363 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.628601 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.628602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.628602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.628742 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.629139 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:41.629155 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:41.629328 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.629435 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.629762 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260927 03:57:41.630573 19609 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40981
I20260927 03:57:41.630672 19609 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:41.630928 19609 heartbeater.cc:511] Master 127.1.50.126:40981 requested a full tablet report, sending...
I20260927 03:57:41.631307 19284 ts_manager.cc:194] Registered new tserver with Master: 419bef7c1dbc468c803f29a89a295646 (127.1.50.66:38925)
I20260927 03:57:41.631804 19284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:52399
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:57:41.657869 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:41.743261 19614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:41.743567 19614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:41.743609 19614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:41.745798 19614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:41.745863 19614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:41.745915 19614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:41.748013 19614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:41.748360 19614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:41.748637 19614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:41.753676 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.753696 19622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:41.753928 19620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:41.754478 19614 server_base.cc:1034] running on GCE node
I20260927 03:57:41.754696 19614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:41.754958 19614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:41.756105 19614 hybrid_clock.cc:648] HybridClock initialized: now 1790481461756106 us; error 52 us; skew 500 ppm
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:41.759142 19614 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:57:41.759670 19614 webserver.cc:460] Webserver started at http://127.1.50.67:36059/ using document root <none> and password file <none>
I20260927 03:57:41.759909 19614 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:41.759961 19614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:41.760092 19614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:41.761274 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "27bd19cd37b94aad981251728875c9e9"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "14616843a71dfcd4c98b0b6088689ee1"
server_key_iv: "605a38dd6f6df1808a2a28a72516a159"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.761662 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "27bd19cd37b94aad981251728875c9e9"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "14616843a71dfcd4c98b0b6088689ee1"
server_key_iv: "605a38dd6f6df1808a2a28a72516a159"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.763293 19614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:57:41.764213 19629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.764369 19614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:41.764501 19614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "27bd19cd37b94aad981251728875c9e9"
format_stamp: "Formatted at 2026-09-27 03:57:41 on dist-test-slave-96fg"
server_key: "14616843a71dfcd4c98b0b6088689ee1"
server_key_iv: "605a38dd6f6df1808a2a28a72516a159"
server_key_version: "encryptionkey@0"
I20260927 03:57:41.764576 19614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:41.804580 19614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:41.806751 19614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:41.806869 19614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:41.807104 19614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:41.807381 19614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:41.807412 19614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.807433 19614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:41.807456 19614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:41.812870 19614 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:36383
I20260927 03:57:41.812929 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:36383 every 8 connection(s)
I20260927 03:57:41.813215 19614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:41.813709 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19614
I20260927 03:57:41.813781 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:41.813906 1225 external_mini_cluster.cc:1428] Setting key 3e4b42698d37d6fee3a1214aa242b4cb
Sep 27 03:57:41 dist-test-slave-96fg krb5kdc[19239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:41.822468 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.814303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:57363 (local address 127.1.50.126:40981)
0927 03:57:41.814345 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.814347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.814589 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:57:41.815194 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.815195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.815212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.815261 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:41.816222 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.816720 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.817696 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.817886 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.817940 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.819989 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:41.819991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:41.819991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:41.819998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:41.821145 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.821537 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.821537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.821538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.821585 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.821847 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.821847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.821848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.821942 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:41.821984 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.822334 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.822399 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:57:41.822688 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.814303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40981 (local address 127.1.50.67:57363)
0927 03:57:41.814456 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.814506 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.815092 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.815332 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.815336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.815929 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:41.816131 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.816134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.816802 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.816804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.817617 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.817620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.817955 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.817956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.818118 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.818607 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:41.818615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:41.819908 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:41.821237 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.821243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.821244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.821455 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.821667 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:41.821668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:41.821668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:41.821785 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:41.822060 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:41.822068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:41.822200 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.822299 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.822472 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260927 03:57:41.823173 19743 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40981
I20260927 03:57:41.823266 19743 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:41.823431 19743 heartbeater.cc:511] Master 127.1.50.126:40981 requested a full tablet report, sending...
I20260927 03:57:41.823764 19284 ts_manager.cc:194] Registered new tserver with Master: 27bd19cd37b94aad981251728875c9e9 (127.1.50.67:36383)
I20260927 03:57:41.824134 19284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:57363
I20260927 03:57:41.825783 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:41.832667 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.827079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.1.50.126:40981)
0927 03:57:41.827139 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.827141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.827263 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:57:41.827300 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.827300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.827308 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.827350 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:41.827810 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.828298 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.829012 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.829205 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.829258 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.830634 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:41.830636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:41.830636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:41.830643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:41.831734 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.832102 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.832103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.832103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.832149 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:41.832283 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:41.832284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:41.832284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:41.832398 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:41.832470 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.832554 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.832603 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:57:41.834249 19284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:41.835816 19284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:41.841967 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.838318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.1.50.66:38925)
0927 03:57:41.838489 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.838494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.838503 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:41.838567 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.838569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.838581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.838650 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:41.838947 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.839693 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841078 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.841589 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841664 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.841726 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.841845 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.841885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260927 03:57:41.841957 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.838007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38925 (local address 127.0.0.1:42946)
0927 03:57:41.838401 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.838410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.838457 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.838800 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.838800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.838802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:41.838832 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.838833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.839784 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.839784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.840978 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.840980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.841672 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.841749 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.841762 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.841826 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.841881 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":171,"threads_started":1}
I20260927 03:57:41.842219 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.837997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:40121 (local address 127.0.0.1:43012)
0927 03:57:41.838196 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.838234 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.838298 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.838660 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.838664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.838666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:41.838734 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.838735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.840047 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.840048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.841116 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.841118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.841972 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.842061 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.842065 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.842097 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.842138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":152,"threads_started":1}
I20260927 03:57:41.842226 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.837997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36383 (local address 127.0.0.1:43282)
0927 03:57:41.838358 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:41.838369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:41.838421 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:41.838941 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:41.838942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:41.838945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:41.839079 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.839081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.840234 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.840235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.841064 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:41.841066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.842003 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.842004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:41.842086 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.842088 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:57:41.842130 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:57:41.842163 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260927 03:57:41.842370 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.838219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.1.50.67:36383)
0927 03:57:41.838747 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.838750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.838756 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:57:41.838786 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.838787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.838801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.838868 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:41.839159 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.840154 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841134 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.841855 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841922 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.841976 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.842260 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.842300 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":90,"threads_started":1}
I20260927 03:57:41.842556 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:41.838206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.1.50.65:40121)
0927 03:57:41.838521 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:41.838525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:41.838535 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:57:41.838566 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:41.838567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:41.838578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:41.838638 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:41.838831 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.839933 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.841175 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:41.841855 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:41.842340 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:41.842413 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:41.842439 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:57:41.842482 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":75,"threads_started":1}
I20260927 03:57:41.843042 19677 tablet_service.cc:1467] Processing CreateTablet for tablet 3266c794cba341e6ad1edf63e360a076 (DEFAULT_TABLE table=test-table [id=e34a858dd8e0467abd287c35d428c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:41.843200 19543 tablet_service.cc:1467] Processing CreateTablet for tablet 3266c794cba341e6ad1edf63e360a076 (DEFAULT_TABLE table=test-table [id=e34a858dd8e0467abd287c35d428c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:41.843179 19409 tablet_service.cc:1467] Processing CreateTablet for tablet 3266c794cba341e6ad1edf63e360a076 (DEFAULT_TABLE table=test-table [id=e34a858dd8e0467abd287c35d428c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:41.843379 19677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3266c794cba341e6ad1edf63e360a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:41.843495 19409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3266c794cba341e6ad1edf63e360a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:41.843495 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3266c794cba341e6ad1edf63e360a076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:41.846005 19760 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Bootstrap starting.
I20260927 03:57:41.846645 19762 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Bootstrap starting.
I20260927 03:57:41.846837 19760 tablet_bootstrap.cc:654] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:41.846849 19761 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Bootstrap starting.
I20260927 03:57:41.847393 19760 log.cc:824] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:41.847537 19761 tablet_bootstrap.cc:654] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:41.848037 19761 log.cc:824] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:41.848161 19760 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: No bootstrap required, opened a new log
I20260927 03:57:41.848217 19760 ts_tablet_manager.cc:1397] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:41.848709 19761 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: No bootstrap required, opened a new log
I20260927 03:57:41.848767 19761 ts_tablet_manager.cc:1397] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:41.848836 19762 tablet_bootstrap.cc:654] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:41.849359 19762 log.cc:824] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:41.850291 19762 tablet_bootstrap.cc:492] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: No bootstrap required, opened a new log
I20260927 03:57:41.850204 19760 raft_consensus.cc:348] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.850389 19760 raft_consensus.cc:374] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:41.850395 19762 ts_tablet_manager.cc:1397] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:57:41.850430 19760 raft_consensus.cc:729] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bd19cd37b94aad981251728875c9e9, State: Initialized, Role: FOLLOWER
I20260927 03:57:41.850559 19760 consensus_queue.cc:260] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.850695 19761 raft_consensus.cc:348] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.850822 19760 ts_tablet_manager.cc:1428] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:41.850847 19761 raft_consensus.cc:374] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:41.850879 19761 raft_consensus.cc:729] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419bef7c1dbc468c803f29a89a295646, State: Initialized, Role: FOLLOWER
I20260927 03:57:41.850932 19743 heartbeater.cc:503] Master 127.1.50.126:40981 was elected leader, sending a full tablet report...
I20260927 03:57:41.850989 19761 consensus_queue.cc:260] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.851248 19761 ts_tablet_manager.cc:1428] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:57:41.851266 19609 heartbeater.cc:503] Master 127.1.50.126:40981 was elected leader, sending a full tablet report...
I20260927 03:57:41.852495 19762 raft_consensus.cc:348] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.852622 19762 raft_consensus.cc:374] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:41.852641 19762 raft_consensus.cc:729] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d608b7ba1f7c4b3199b501b19ad23baf, State: Initialized, Role: FOLLOWER
I20260927 03:57:41.852720 19762 consensus_queue.cc:260] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:41.852906 19762 ts_tablet_manager.cc:1428] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:57:41.852964 19475 heartbeater.cc:503] Master 127.1.50.126:40981 was elected leader, sending a full tablet report...
W20260927 03:57:41.870456 19610 tablet.cc:2369] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:41.973937 19476 tablet.cc:2369] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:42.063746 19744 tablet.cc:2369] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:42.118824 19768 raft_consensus.cc:482] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:42.119009 19768 raft_consensus.cc:504] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:42.119446 19768 leader_election.cc:288] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27bd19cd37b94aad981251728875c9e9 (127.1.50.67:36383), 419bef7c1dbc468c803f29a89a295646 (127.1.50.66:38925)
I20260927 03:57:42.122865 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.119772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:33021 (local address 127.1.50.66:38925)
0927 03:57:42.119822 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.119824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.119986 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:57:42.120107 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.120108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.120121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.120166 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:42.120387 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.120913 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122073 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.122520 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122584 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.122595 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.122741 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.122782 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:57:42.122911 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.119862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:37369 (local address 127.1.50.67:36383)
0927 03:57:42.119920 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.119922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.120006 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:57:42.120078 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.120079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.120091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.120141 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:42.120346 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.120861 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122068 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.122552 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122622 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.122636 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.122783 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.122825 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:57:42.123121 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.119700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38925 (local address 127.1.50.65:33021)
0927 03:57:42.119888 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.119899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.119987 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.120183 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.120184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.120187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:42.120317 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.120321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.120991 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.120999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.121954 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.121956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.122600 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.122677 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.122682 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.122729 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.122786 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
I20260927 03:57:42.123147 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.119708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:36383 (local address 127.1.50.65:37369)
0927 03:57:42.119919 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.119931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.120007 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.120153 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.120154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.120157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:42.120277 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.120280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.120938 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.120939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.121931 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.121934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.122634 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.122635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.122717 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.122722 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.122770 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.122827 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":74,"thread_start_us":123,"threads_started":1}
I20260927 03:57:42.123606 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3266c794cba341e6ad1edf63e360a076" candidate_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419bef7c1dbc468c803f29a89a295646" is_pre_election: true
I20260927 03:57:42.123588 19697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3266c794cba341e6ad1edf63e360a076" candidate_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bd19cd37b94aad981251728875c9e9" is_pre_election: true
I20260927 03:57:42.123783 19563 raft_consensus.cc:2454] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d608b7ba1f7c4b3199b501b19ad23baf in term 0.
I20260927 03:57:42.123783 19697 raft_consensus.cc:2454] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d608b7ba1f7c4b3199b501b19ad23baf in term 0.
I20260927 03:57:42.124049 19363 leader_election.cc:302] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419bef7c1dbc468c803f29a89a295646, d608b7ba1f7c4b3199b501b19ad23baf; no voters:
I20260927 03:57:42.124224 19768 raft_consensus.cc:2781] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:42.124306 19768 raft_consensus.cc:482] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:42.124323 19768 raft_consensus.cc:3037] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:42.125387 19768 raft_consensus.cc:504] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:42.125574 19768 leader_election.cc:288] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [CANDIDATE]: Term 1 election: Requested vote from peers 27bd19cd37b94aad981251728875c9e9 (127.1.50.67:36383), 419bef7c1dbc468c803f29a89a295646 (127.1.50.66:38925)
I20260927 03:57:42.125816 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3266c794cba341e6ad1edf63e360a076" candidate_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419bef7c1dbc468c803f29a89a295646"
I20260927 03:57:42.125804 19697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3266c794cba341e6ad1edf63e360a076" candidate_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bd19cd37b94aad981251728875c9e9"
I20260927 03:57:42.125932 19697 raft_consensus.cc:3037] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:42.125932 19563 raft_consensus.cc:3037] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:42.127010 19563 raft_consensus.cc:2454] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d608b7ba1f7c4b3199b501b19ad23baf in term 1.
I20260927 03:57:42.127010 19697 raft_consensus.cc:2454] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d608b7ba1f7c4b3199b501b19ad23baf in term 1.
I20260927 03:57:42.127269 19362 leader_election.cc:302] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bd19cd37b94aad981251728875c9e9, d608b7ba1f7c4b3199b501b19ad23baf; no voters:
I20260927 03:57:42.127430 19768 raft_consensus.cc:2781] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:42.127676 19768 raft_consensus.cc:686] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [term 1 LEADER]: Becoming Leader. State: Replica: d608b7ba1f7c4b3199b501b19ad23baf, State: Running, Role: LEADER
I20260927 03:57:42.127802 19768 consensus_queue.cc:237] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } }
I20260927 03:57:42.129086 19284 catalog_manager.cc:5495] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf reported cstate change: term changed from 0 to 1, leader changed from <none> to d608b7ba1f7c4b3199b501b19ad23baf (127.1.50.65). New cstate: current_term: 1 leader_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d608b7ba1f7c4b3199b501b19ad23baf" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 40121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 } health_report { overall_health: UNKNOWN } } }
I20260927 03:57:42.201007 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.198193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.1.50.65:40121)
0927 03:57:42.198267 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.198269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.198281 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:57:42.198354 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.198355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.198379 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.198452 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:42.198636 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.199114 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.199999 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.200248 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.200395 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.200605 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:42.200700 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:42.200800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.200884 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.200912 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:57:42.203387 19563 raft_consensus.cc:1261] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 1 FOLLOWER]: Refusing update from remote peer d608b7ba1f7c4b3199b501b19ad23baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:42.203548 19697 raft_consensus.cc:1261] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 1 FOLLOWER]: Refusing update from remote peer d608b7ba1f7c4b3199b501b19ad23baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:42.203871 19771 consensus_queue.cc:1035] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "419bef7c1dbc468c803f29a89a295646" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 38925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:42.204102 19768 consensus_queue.cc:1035] T 3266c794cba341e6ad1edf63e360a076 P d608b7ba1f7c4b3199b501b19ad23baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bd19cd37b94aad981251728875c9e9" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:42.207448 19775 mvcc.cc:205] Tried to move back new op lower bound from 7333812069181489152 to 7333812068876472320. Current Snapshot: MvccSnapshot[applied={T|T < 7333812069181489152}]
I20260927 03:57:42.207621 19777 mvcc.cc:205] Tried to move back new op lower bound from 7333812069181489152 to 7333812068876472320. Current Snapshot: MvccSnapshot[applied={T|T < 7333812069181489152}]
I20260927 03:57:42.210037 19284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:42.210172 19284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:42.211133 19284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b220c65f151541a8968c17aca2c0467d: Sending DeleteTablet for 3 replicas of tablet 3266c794cba341e6ad1edf63e360a076
I20260927 03:57:42.211586 19677 tablet_service.cc:1514] Processing DeleteTablet for tablet 3266c794cba341e6ad1edf63e360a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:42 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:43282
I20260927 03:57:42.211568 19543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3266c794cba341e6ad1edf63e360a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:42 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:42946
I20260927 03:57:42.211980 19786 tablet_replica.cc:330] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: stopping tablet replica
I20260927 03:57:42.211993 19409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3266c794cba341e6ad1edf63e360a076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:42 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:43012
I20260927 03:57:42.212241 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19346
I20260927 03:57:42.214349 19786 raft_consensus.cc:2229] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:42.214520 19786 raft_consensus.cc:2258] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:42.214713 19785 tablet_replica.cc:330] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: stopping tablet replica
I20260927 03:57:42.214859 19785 raft_consensus.cc:2229] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:42.214978 19785 raft_consensus.cc:2258] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:42.215204 19786 ts_tablet_manager.cc:1905] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:42.215559 19785 ts_tablet_manager.cc:1905] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:42.217438 19786 ts_tablet_manager.cc:1918] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:42.217532 19786 log.cc:1196] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/3266c794cba341e6ad1edf63e360a076
W20260927 03:57:42.217514 19271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:40121 (error 108)
I20260927 03:57:42.217621 19785 ts_tablet_manager.cc:1918] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:42.217685 19785 log.cc:1196] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/3266c794cba341e6ad1edf63e360a076
I20260927 03:57:42.217850 19786 ts_tablet_manager.cc:1939] T 3266c794cba341e6ad1edf63e360a076 P 419bef7c1dbc468c803f29a89a295646: Deleting consensus metadata
I20260927 03:57:42.217928 19785 ts_tablet_manager.cc:1939] T 3266c794cba341e6ad1edf63e360a076 P 27bd19cd37b94aad981251728875c9e9: Deleting consensus metadata
I20260927 03:57:42.218156 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19480
I20260927 03:57:42.219887 19268 catalog_manager.cc:4841] TS 27bd19cd37b94aad981251728875c9e9 (127.1.50.67:36383): tablet 3266c794cba341e6ad1edf63e360a076 (table test-table [id=e34a858dd8e0467abd287c35d428c181]) successfully deleted
I20260927 03:57:42.222677 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.222473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:40121 (local address 127.0.0.1:43042)
0927 03:57:42.222551 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.222622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 03:57:42.222939 19271 catalog_manager.cc:4584] TS d608b7ba1f7c4b3199b501b19ad23baf (127.1.50.65:40121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3266c794cba341e6ad1edf63e360a076: Network error: Client connection negotiation failed: client connection to 127.1.50.65:40121: connect: Connection refused (error 111)
I20260927 03:57:42.223757 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.223630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:38925 (local address 127.0.0.1:42950)
0927 03:57:42.223691 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.223723 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.66:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:57:42.223753 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19614
I20260927 03:57:42.229159 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19252
2026-09-27T03:57:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2186 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T03:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:42Z Disabled control of system clock
I20260927 03:57:42.258751 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40983
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:42.349342 19792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:42.349629 19792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:42.349653 19792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:42.351002 19792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:42.351050 19792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:42.351061 19792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:42.351073 19792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:42.351084 19792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:42.352551 19792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:42.352811 19792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:42.353140 19792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:42.358563 19798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.358582 19797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.358541 19800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.359225 19792 server_base.cc:1034] running on GCE node
I20260927 03:57:42.359531 19792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:42.359854 19792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:42.361006 19792 hybrid_clock.cc:648] HybridClock initialized: now 1790481462360975 us; error 60 us; skew 500 ppm
I20260927 03:57:42.367184 19792 webserver.cc:460] Webserver started at http://127.1.50.126:41219/ using document root <none> and password file <none>
I20260927 03:57:42.367457 19792 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:42.367511 19792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:42.367640 19792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:42.368757 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "f6e6197eea9146398bda083973f2553b"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "2b078c6050235c3d4f8b8c2843467a24"
server_key_iv: "f99b298b3aae6460c2d30ec83b0b0277"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.369063 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "f6e6197eea9146398bda083973f2553b"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "2b078c6050235c3d4f8b8c2843467a24"
server_key_iv: "f99b298b3aae6460c2d30ec83b0b0277"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.370487 19792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:57:42.371443 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.371785 19792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:42.371899 19792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "f6e6197eea9146398bda083973f2553b"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "2b078c6050235c3d4f8b8c2843467a24"
server_key_iv: "f99b298b3aae6460c2d30ec83b0b0277"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.371978 19792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:42.385443 19792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:42.385780 19792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:42.385912 19792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:42.390101 19792 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:40983
I20260927 03:57:42.390188 19858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:40983 every 8 connection(s)
I20260927 03:57:42.390517 19792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:42.391660 19859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:42.394276 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19792
I20260927 03:57:42.394383 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:42.394518 19859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: Bootstrap starting.
I20260927 03:57:42.394590 1225 external_mini_cluster.cc:1428] Setting key 012da64a7a09761765a1a602696c500e
I20260927 03:57:42.395241 19859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:42.395587 19859 log.cc:824] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:42.396466 19859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: No bootstrap required, opened a new log
I20260927 03:57:42.398679 19859 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } }
I20260927 03:57:42.398859 19859 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:42.398883 19859 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e6197eea9146398bda083973f2553b, State: Initialized, Role: FOLLOWER
I20260927 03:57:42.399015 19859 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } }
I20260927 03:57:42.399093 19859 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:42.399117 19859 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:42.399168 19859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:42.399888 19859 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } }
I20260927 03:57:42.400010 19859 leader_election.cc:302] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e6197eea9146398bda083973f2553b; no voters:
I20260927 03:57:42.400266 19859 leader_election.cc:288] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:42.400321 19864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:42.400445 19864 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [term 1 LEADER]: Becoming Leader. State: Replica: f6e6197eea9146398bda083973f2553b, State: Running, Role: LEADER
I20260927 03:57:42.400550 19864 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } }
I20260927 03:57:42.400600 19859 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:42.400979 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6e6197eea9146398bda083973f2553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } } }
I20260927 03:57:42.401225 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.395233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.1.50.126:40983)
0927 03:57:42.395524 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.395534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.395655 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:57:42.396457 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.396468 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.396558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.396714 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:42.396942 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.397876 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.398692 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.399254 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.399381 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.400272 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:42.400277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:42.400277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:42.400545 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:42.400601 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:42.400617 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:42.400850 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.400887 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.400982 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":134,"threads_started":1}
I20260927 03:57:42.401250 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:42.401732 19865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6e6197eea9146398bda083973f2553b. Latest consensus state: current_term: 1 leader_uuid: "f6e6197eea9146398bda083973f2553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e6197eea9146398bda083973f2553b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40983 } } }
I20260927 03:57:42.401818 19865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:42.402069 19873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:42.402812 19873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:42.404917 19873 catalog_manager.cc:1347] Generated new cluster ID: b5f9563387b54ad7b3c4cd2658a9f427
I20260927 03:57:42.404978 19873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:42.412025 19873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:42.412636 19873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:42.418362 19873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: Generated new TSK 0
I20260927 03:57:42.418582 19873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:42.424084 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--builtin_ntp_servers=127.1.50.84:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:42.513291 19883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:42.513532 19883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:42.513556 19883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:42.514870 19883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:42.514931 19883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:42.514966 19883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:42.516427 19883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:42.516678 19883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:42.517019 19883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:42.523898 19888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.524175 19883 server_base.cc:1034] running on GCE node
W20260927 03:57:42.524116 19891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.523923 19889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.524549 19883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:42.524807 19883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:42.525977 19883 hybrid_clock.cc:648] HybridClock initialized: now 1790481462525959 us; error 33 us; skew 500 ppm
I20260927 03:57:42.527338 19883 webserver.cc:460] Webserver started at http://127.1.50.65:35745/ using document root <none> and password file <none>
I20260927 03:57:42.527580 19883 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:42.527630 19883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:42.527767 19883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:42.528854 19883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "3d3e534e9b1d4c4c84f690b0a75f0571"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "0c816d7d407431df1134d00d9937508e"
server_key_iv: "aefdde5f0ebef22ff7196032d96b526b"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.529188 19883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "3d3e534e9b1d4c4c84f690b0a75f0571"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "0c816d7d407431df1134d00d9937508e"
server_key_iv: "aefdde5f0ebef22ff7196032d96b526b"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.530592 19883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:57:42.531337 19897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.531482 19883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:57:42.531571 19883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "3d3e534e9b1d4c4c84f690b0a75f0571"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "0c816d7d407431df1134d00d9937508e"
server_key_iv: "aefdde5f0ebef22ff7196032d96b526b"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.531649 19883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:42.560256 19883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:42.560564 19883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:42.560684 19883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:42.560930 19883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:42.561262 19883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:42.561296 19883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.561326 19883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:42.561350 19883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.567730 19883 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:41373
I20260927 03:57:42.567796 20010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:41373 every 8 connection(s)
I20260927 03:57:42.568107 19883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:42.570186 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 19883
I20260927 03:57:42.570312 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:42.570499 1225 external_mini_cluster.cc:1428] Setting key 26ab47576a5e1bf53b1efa27b31d7aa4
I20260927 03:57:42.571702 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--builtin_ntp_servers=127.1.50.84:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:57:42.574370 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.568914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:36307 (local address 127.1.50.126:40983)
0927 03:57:42.568981 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.568983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.569167 (+ 184us) server_negotiation.cc:407] Connection header received
0927 03:57:42.569665 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.569667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.569679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.569766 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:42.570629 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.571241 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.572628 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.572852 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.572943 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.574073 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:42.574074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:42.574074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:42.574079 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:42.574093 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:42.574097 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:42.574142 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.574233 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.574264 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:57:42.574617 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.568849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40983 (local address 127.1.50.65:36307)
0927 03:57:42.569032 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.569081 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.569550 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.569822 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.569825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.569833 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:42.570515 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.570522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.571342 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.571343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.572513 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.572516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.572979 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.572982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.573167 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.573873 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:42.573874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:42.573910 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:42.573941 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:42.574151 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:42.574158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:42.574162 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.574230 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.574365 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260927 03:57:42.575314 20011 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40983
I20260927 03:57:42.575435 20011 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:42.575663 20011 heartbeater.cc:511] Master 127.1.50.126:40983 requested a full tablet report, sending...
I20260927 03:57:42.576378 19823 ts_manager.cc:194] Registered new tserver with Master: 3d3e534e9b1d4c4c84f690b0a75f0571 (127.1.50.65:41373)
I20260927 03:57:42.577335 19823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.65:36307
W20260927 03:57:42.653007 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:42.653263 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:42.653287 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:42.654505 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:42.654561 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:42.654589 20014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:42.656239 20014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:42.656489 20014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:42.656782 20014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:42.662470 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.662485 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.662611 20014 server_base.cc:1034] running on GCE node
W20260927 03:57:42.662503 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.663015 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:42.663303 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:42.664496 20014 hybrid_clock.cc:648] HybridClock initialized: now 1790481462664472 us; error 45 us; skew 500 ppm
I20260927 03:57:42.666168 20014 webserver.cc:460] Webserver started at http://127.1.50.66:43153/ using document root <none> and password file <none>
I20260927 03:57:42.666455 20014 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:42.666508 20014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:42.666631 20014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:42.668104 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "e0f122edc3084a66a9eeb36b6705a010"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "823a2fa1a649dd5dc1d5123353e7c33d"
server_key_iv: "c07435313b4258bbab8810f749e16416"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.668497 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "e0f122edc3084a66a9eeb36b6705a010"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "823a2fa1a649dd5dc1d5123353e7c33d"
server_key_iv: "c07435313b4258bbab8810f749e16416"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.670058 20014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:57:42.671022 20028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.671276 20014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:42.671379 20014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "e0f122edc3084a66a9eeb36b6705a010"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "823a2fa1a649dd5dc1d5123353e7c33d"
server_key_iv: "c07435313b4258bbab8810f749e16416"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.671461 20014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:42.694609 20014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:42.695021 20014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:42.695168 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:42.695453 20014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:42.695842 20014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:42.695878 20014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.695911 20014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:42.695935 20014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.703225 20014 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:44719
I20260927 03:57:42.703378 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:44719 every 8 connection(s)
I20260927 03:57:42.703672 20014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:57:42.707683 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20014
I20260927 03:57:42.707789 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:42.707983 1225 external_mini_cluster.cc:1428] Setting key a810058b8c63f777ebff381979cde917
I20260927 03:57:42.709365 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--builtin_ntp_servers=127.1.50.84:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:57:42.710500 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.704503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:58811 (local address 127.1.50.126:40983)
0927 03:57:42.704613 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.704616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.705108 (+ 492us) server_negotiation.cc:407] Connection header received
0927 03:57:42.705760 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.705762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.705776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.705831 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:42.706696 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.707254 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.708420 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.708689 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.708763 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.710140 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:42.710142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:42.710142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:42.710148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:42.710171 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:42.710176 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:42.710233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.710388 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.710422 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 03:57:42.710776 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.704466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40983 (local address 127.1.50.66:58811)
0927 03:57:42.704814 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.704860 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.705647 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.705916 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.705920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.705930 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:42.706598 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.706605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.707346 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.707347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.708355 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.708358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.708963 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.708965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.709236 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.709882 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:42.709883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:42.709931 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:42.709969 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:42.710255 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:42.710261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:42.710264 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.710349 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.710494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":123,"threads_started":1}
I20260927 03:57:42.711406 20142 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40983
I20260927 03:57:42.711500 20142 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:42.711699 20142 heartbeater.cc:511] Master 127.1.50.126:40983 requested a full tablet report, sending...
I20260927 03:57:42.712078 19823 ts_manager.cc:194] Registered new tserver with Master: e0f122edc3084a66a9eeb36b6705a010 (127.1.50.66:44719)
I20260927 03:57:42.712457 19823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.66:58811
W20260927 03:57:42.803838 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:42.804107 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:42.804141 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:42.805446 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:42.805486 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:42.805522 20145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:42.807142 20145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:42.807432 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:42.807776 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:42.813712 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.813712 20150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:42.813733 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:42.814371 20145 server_base.cc:1034] running on GCE node
I20260927 03:57:42.814723 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:42.815076 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:42.816255 20145 hybrid_clock.cc:648] HybridClock initialized: now 1790481462816232 us; error 59 us; skew 500 ppm
I20260927 03:57:42.817730 20145 webserver.cc:460] Webserver started at http://127.1.50.67:35015/ using document root <none> and password file <none>
I20260927 03:57:42.818105 20145 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:42.818161 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:42.818301 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:42.819519 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "a39dabeec4584301ad4668f9fe051a7f"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "f407d062bc6560ee674239f33ce69a53"
server_key_iv: "9ffa0caa1da962fea91d3225d2a05e14"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.819897 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "a39dabeec4584301ad4668f9fe051a7f"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "f407d062bc6560ee674239f33ce69a53"
server_key_iv: "9ffa0caa1da962fea91d3225d2a05e14"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.821456 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:42.822630 20159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.822882 20145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:42.822986 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "a39dabeec4584301ad4668f9fe051a7f"
format_stamp: "Formatted at 2026-09-27 03:57:42 on dist-test-slave-96fg"
server_key: "f407d062bc6560ee674239f33ce69a53"
server_key_iv: "9ffa0caa1da962fea91d3225d2a05e14"
server_key_version: "encryptionkey@0"
I20260927 03:57:42.823062 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:42.837150 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:42.837564 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:42.837716 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:42.838065 20145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:42.838477 20145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:42.838513 20145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.838547 20145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:42.838570 20145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:42.845759 20145 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:35261
I20260927 03:57:42.845804 20272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:35261 every 8 connection(s)
I20260927 03:57:42.846241 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:42.852593 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.847134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:45335 (local address 127.1.50.126:40983)
0927 03:57:42.847206 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.847208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.847508 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:57:42.848130 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.848132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.848146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.848201 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:42.848965 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.849657 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.850967 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.851168 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.851226 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.852304 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:42.852306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:42.852306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:42.852313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:42.852330 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:42.852334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:42.852378 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.852461 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.852495 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:57:42.852865 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.847099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40983 (local address 127.1.50.67:45335)
0927 03:57:42.847352 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.847395 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.848022 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.848286 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.848290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.848300 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:42.848875 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.848880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.849854 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.849856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.850840 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.850843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.851244 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.851245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.851398 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.852129 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:42.852129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:42.852164 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:42.852194 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:42.852387 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:42.852392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:42.852396 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.852462 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.852590 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260927 03:57:42.853499 20273 heartbeater.cc:347] Connected to a master server at 127.1.50.126:40983
I20260927 03:57:42.853633 20273 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:42.853952 20273 heartbeater.cc:511] Master 127.1.50.126:40983 requested a full tablet report, sending...
I20260927 03:57:42.854529 19823 ts_manager.cc:194] Registered new tserver with Master: a39dabeec4584301ad4668f9fe051a7f (127.1.50.67:35261)
I20260927 03:57:42.854904 19823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.67:45335
I20260927 03:57:42.856074 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20145
I20260927 03:57:42.856169 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:42.856329 1225 external_mini_cluster.cc:1428] Setting key de2dfa48964f4ac44d6813d916ccb079
I20260927 03:57:42.857649 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:42.862672 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.859299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.1.50.126:40983)
0927 03:57:42.859375 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.859377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.859591 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:57:42.859642 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.859643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.859654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.859709 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:42.859885 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.860440 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.861264 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.861472 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.861541 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.862390 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:42.862391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:42.862392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:42.862397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:42.862414 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:42.862418 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:42.862495 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.862560 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.862572 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:57:42.864758 19823 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:42.866587 19823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:42.873682 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:35261 (local address 127.0.0.1:60930)
0927 03:57:42.869548 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.869557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.869603 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.870048 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.870049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.870051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:42.870097 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.870098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871577 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.871578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.872602 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.872605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.873336 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.873423 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.873439 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.873516 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.873577 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":103,"thread_start_us":125,"threads_started":1}
I20260927 03:57:42.873692 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.1.50.67:35261)
0927 03:57:42.869790 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.869793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.869804 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:42.869843 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.869845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.869859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.869944 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:42.870208 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871484 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.872696 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.873246 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.873430 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.873530 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.873582 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":97,"threads_started":1}
I20260927 03:57:42.874418 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.1.50.65:41373)
0927 03:57:42.869715 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.869720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.870173 (+ 453us) server_negotiation.cc:407] Connection header received
0927 03:57:42.870293 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.870295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.870311 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.870362 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:42.870648 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871415 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873059 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.873799 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873866 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.873962 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.874243 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.874320 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260927 03:57:42.874889 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.1.50.66:44719)
0927 03:57:42.869715 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:42.869720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:42.869734 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:57:42.869778 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:42.869780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:42.869797 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:42.869900 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:42.870209 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871457 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873260 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.874170 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.874252 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.874338 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:42.874738 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 03:57:42.874795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":101,"threads_started":1}
I20260927 03:57:42.874830 20207 tablet_service.cc:1467] Processing CreateTablet for tablet abe532c3cff941f7af6b59724fac279d (DEFAULT_TABLE table=test-table [id=b16dd7150b2f46538dd36b738ab7f7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:42.874919 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:44719 (local address 127.0.0.1:36756)
0927 03:57:42.869453 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.869508 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.869600 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.869966 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.869971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.869974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:42.870096 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.870098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871542 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.871543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.873159 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.873160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.874378 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.874379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.874453 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.874458 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.874718 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.874786 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":106,"thread_start_us":66,"threads_started":1}
I20260927 03:57:42.875278 20207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe532c3cff941f7af6b59724fac279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:42.878324 20287 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Bootstrap starting.
I20260927 03:57:42.878413 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:42.869387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41373 (local address 127.0.0.1:46730)
0927 03:57:42.870059 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:42.870071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:42.870204 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:42.870496 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:42.870497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:42.870499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:42.870559 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.870561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.871489 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.871490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.872966 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:42.872969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:42.873991 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:42.873992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:42.874074 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:42.874079 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:42.874196 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:57:42.874242 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":83,"thread_start_us":77,"threads_started":1}
I20260927 03:57:42.878895 20076 tablet_service.cc:1467] Processing CreateTablet for tablet abe532c3cff941f7af6b59724fac279d (DEFAULT_TABLE table=test-table [id=b16dd7150b2f46538dd36b738ab7f7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:42.879097 20287 tablet_bootstrap.cc:654] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:42.879233 20076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe532c3cff941f7af6b59724fac279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:42.879431 20287 log.cc:824] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:42.880335 20287 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: No bootstrap required, opened a new log
I20260927 03:57:42.880424 20287 ts_tablet_manager.cc:1397] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:42.882102 20289 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Bootstrap starting.
I20260927 03:57:42.882797 20289 tablet_bootstrap.cc:654] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:42.882912 20287 raft_consensus.cc:348] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.883096 20287 raft_consensus.cc:374] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:42.883087 20289 log.cc:824] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:42.883131 20287 raft_consensus.cc:729] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39dabeec4584301ad4668f9fe051a7f, State: Initialized, Role: FOLLOWER
I20260927 03:57:42.883271 20287 consensus_queue.cc:260] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.883145 19945 tablet_service.cc:1467] Processing CreateTablet for tablet abe532c3cff941f7af6b59724fac279d (DEFAULT_TABLE table=test-table [id=b16dd7150b2f46538dd36b738ab7f7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:42.883589 19945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe532c3cff941f7af6b59724fac279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:42.883595 20287 ts_tablet_manager.cc:1428] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 03:57:42.883648 20273 heartbeater.cc:503] Master 127.1.50.126:40983 was elected leader, sending a full tablet report...
I20260927 03:57:42.884061 20289 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: No bootstrap required, opened a new log
I20260927 03:57:42.884130 20289 ts_tablet_manager.cc:1397] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:42.886376 20289 raft_consensus.cc:348] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.886550 20289 raft_consensus.cc:374] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:42.886575 20289 raft_consensus.cc:729] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f122edc3084a66a9eeb36b6705a010, State: Initialized, Role: FOLLOWER
I20260927 03:57:42.886726 20289 consensus_queue.cc:260] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.887045 20289 ts_tablet_manager.cc:1428] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:42.887105 20142 heartbeater.cc:503] Master 127.1.50.126:40983 was elected leader, sending a full tablet report...
I20260927 03:57:42.887216 20292 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Bootstrap starting.
I20260927 03:57:42.887917 20292 tablet_bootstrap.cc:654] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:42.888219 20292 log.cc:824] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:42.889060 20292 tablet_bootstrap.cc:492] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: No bootstrap required, opened a new log
I20260927 03:57:42.889127 20292 ts_tablet_manager.cc:1397] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:42.891150 20292 raft_consensus.cc:348] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.891304 20292 raft_consensus.cc:374] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:42.891325 20292 raft_consensus.cc:729] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3e534e9b1d4c4c84f690b0a75f0571, State: Initialized, Role: FOLLOWER
I20260927 03:57:42.891438 20292 consensus_queue.cc:260] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:42.891716 20292 ts_tablet_manager.cc:1428] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:57:42.891842 20011 heartbeater.cc:503] Master 127.1.50.126:40983 was elected leader, sending a full tablet report...
W20260927 03:57:42.954397 20143 tablet.cc:2369] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:43.068876 20012 tablet.cc:2369] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:43.096964 20274 tablet.cc:2369] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:43.114401 20295 raft_consensus.cc:482] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:43.114596 20295 raft_consensus.cc:504] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:43.115037 20295 leader_election.cc:288] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0f122edc3084a66a9eeb36b6705a010 (127.1.50.66:44719), a39dabeec4584301ad4668f9fe051a7f (127.1.50.67:35261)
I20260927 03:57:43.118517 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.115399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:47969 (local address 127.1.50.67:35261)
0927 03:57:43.115492 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:43.115494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:43.115598 (+ 104us) server_negotiation.cc:407] Connection header received
0927 03:57:43.115667 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:43.115669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:43.115680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:43.115765 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:43.116026 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.116549 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.117719 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.118147 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.118227 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:43.118240 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:43.118388 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:57:43.118430 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:57:43.118536 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.115402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:47651 (local address 127.1.50.66:44719)
0927 03:57:43.115495 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:43.115497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:43.115585 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:57:43.115624 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:43.115626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:43.115636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:43.115691 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:43.115927 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.116452 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.117617 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.118136 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.118199 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:43.118215 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:43.118408 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 03:57:43.118459 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 03:57:43.118639 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.115286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:35261 (local address 127.1.50.65:47969)
0927 03:57:43.115495 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:43.115509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:43.115601 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:43.115780 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:43.115781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:43.115784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:43.115907 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:43.115910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.116679 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.116680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:43.117597 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:43.117600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.118250 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.118251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:43.118314 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:43.118319 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:43.118362 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:57:43.118434 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":27,"thread_start_us":57,"threads_started":1}
I20260927 03:57:43.118645 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.115263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:44719 (local address 127.1.50.65:47651)
0927 03:57:43.115443 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:43.115457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:43.115526 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:43.115717 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:43.115719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:43.115722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:43.115844 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:43.115847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.116545 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.116546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:43.117493 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:43.117495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.118242 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.118244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:43.118321 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:43.118326 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:43.118386 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:57:43.118438 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":51,"thread_start_us":76,"threads_started":1}
I20260927 03:57:43.119128 20227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe532c3cff941f7af6b59724fac279d" candidate_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39dabeec4584301ad4668f9fe051a7f" is_pre_election: true
I20260927 03:57:43.119340 20227 raft_consensus.cc:2454] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3e534e9b1d4c4c84f690b0a75f0571 in term 0.
I20260927 03:57:43.119313 20096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe532c3cff941f7af6b59724fac279d" candidate_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f122edc3084a66a9eeb36b6705a010" is_pre_election: true
I20260927 03:57:43.119500 20096 raft_consensus.cc:2454] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3e534e9b1d4c4c84f690b0a75f0571 in term 0.
I20260927 03:57:43.119665 19901 leader_election.cc:302] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3e534e9b1d4c4c84f690b0a75f0571, a39dabeec4584301ad4668f9fe051a7f; no voters:
I20260927 03:57:43.119807 20295 raft_consensus.cc:2781] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:43.119868 20295 raft_consensus.cc:482] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:43.119892 20295 raft_consensus.cc:3037] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:43.120738 20295 raft_consensus.cc:504] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:43.120991 20295 leader_election.cc:288] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [CANDIDATE]: Term 1 election: Requested vote from peers e0f122edc3084a66a9eeb36b6705a010 (127.1.50.66:44719), a39dabeec4584301ad4668f9fe051a7f (127.1.50.67:35261)
I20260927 03:57:43.121199 20096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe532c3cff941f7af6b59724fac279d" candidate_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f122edc3084a66a9eeb36b6705a010"
I20260927 03:57:43.121266 20227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe532c3cff941f7af6b59724fac279d" candidate_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39dabeec4584301ad4668f9fe051a7f"
I20260927 03:57:43.121297 20096 raft_consensus.cc:3037] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:43.121383 20227 raft_consensus.cc:3037] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:43.122398 20227 raft_consensus.cc:2454] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3e534e9b1d4c4c84f690b0a75f0571 in term 1.
I20260927 03:57:43.122424 20096 raft_consensus.cc:2454] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3e534e9b1d4c4c84f690b0a75f0571 in term 1.
I20260927 03:57:43.122612 19901 leader_election.cc:302] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3e534e9b1d4c4c84f690b0a75f0571, a39dabeec4584301ad4668f9fe051a7f; no voters:
I20260927 03:57:43.122738 20295 raft_consensus.cc:2781] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:43.122903 20295 raft_consensus.cc:686] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3e534e9b1d4c4c84f690b0a75f0571, State: Running, Role: LEADER
I20260927 03:57:43.123013 20295 consensus_queue.cc:237] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } }
I20260927 03:57:43.123919 19823 catalog_manager.cc:5495] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d3e534e9b1d4c4c84f690b0a75f0571 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d3e534e9b1d4c4c84f690b0a75f0571" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 41373 } health_report { overall_health: HEALTHY } } }
I20260927 03:57:43.161010 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.158498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.1.50.65:41373)
0927 03:57:43.158561 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:43.158564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:43.158605 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:57:43.158664 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:43.158665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:43.158688 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:43.158755 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:43.158962 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.159502 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.160331 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:43.160518 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:43.160578 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:43.160715 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:43.160809 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:43.160884 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:43.160937 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:57:43.160948 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:57:43.163327 20227 raft_consensus.cc:1261] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 1 FOLLOWER]: Refusing update from remote peer 3d3e534e9b1d4c4c84f690b0a75f0571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:43.163327 20096 raft_consensus.cc:1261] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 1 FOLLOWER]: Refusing update from remote peer 3d3e534e9b1d4c4c84f690b0a75f0571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:43.163779 20295 consensus_queue.cc:1035] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f122edc3084a66a9eeb36b6705a010" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:43.163992 20298 consensus_queue.cc:1035] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39dabeec4584301ad4668f9fe051a7f" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:43.166888 20301 mvcc.cc:205] Tried to move back new op lower bound from 7333812073113849856 to 7333812072952532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333812073113849856}]
I20260927 03:57:43.169955 19823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:43.170114 19823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:43.170989 19823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6e6197eea9146398bda083973f2553b: Sending DeleteTablet for 3 replicas of tablet abe532c3cff941f7af6b59724fac279d
I20260927 03:57:43.171387 19945 tablet_service.cc:1514] Processing DeleteTablet for tablet abe532c3cff941f7af6b59724fac279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:43 UTC) from {username='slave'} at 127.0.0.1:46730
I20260927 03:57:43.171409 20207 tablet_service.cc:1514] Processing DeleteTablet for tablet abe532c3cff941f7af6b59724fac279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:43 UTC) from {username='slave'} at 127.0.0.1:60930
I20260927 03:57:43.171694 20310 tablet_replica.cc:330] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571: stopping tablet replica
I20260927 03:57:43.171700 20076 tablet_service.cc:1514] Processing DeleteTablet for tablet abe532c3cff941f7af6b59724fac279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:43 UTC) from {username='slave'} at 127.0.0.1:36756
I20260927 03:57:43.171854 20310 raft_consensus.cc:2229] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:57:43.172021 20310 raft_consensus.cc:2258] T abe532c3cff941f7af6b59724fac279d P 3d3e534e9b1d4c4c84f690b0a75f0571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:43.172055 20312 tablet_replica.cc:330] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: stopping tablet replica
I20260927 03:57:43.172190 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19883
I20260927 03:57:43.172240 20312 raft_consensus.cc:2229] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:43.173864 20312 raft_consensus.cc:2258] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:43.174705 20312 ts_tablet_manager.cc:1905] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:43.176765 20312 ts_tablet_manager.cc:1918] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:43.176846 20312 log.cc:1196] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/abe532c3cff941f7af6b59724fac279d
I20260927 03:57:43.177062 20312 ts_tablet_manager.cc:1939] T abe532c3cff941f7af6b59724fac279d P e0f122edc3084a66a9eeb36b6705a010: Deleting consensus metadata
I20260927 03:57:43.177520 19807 catalog_manager.cc:4841] TS e0f122edc3084a66a9eeb36b6705a010 (127.1.50.66:44719): tablet abe532c3cff941f7af6b59724fac279d (table test-table [id=b16dd7150b2f46538dd36b738ab7f7b1]) successfully deleted
W20260927 03:57:43.178571 19807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:41373 (error 108)
I20260927 03:57:43.178750 20311 tablet_replica.cc:330] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: stopping tablet replica
I20260927 03:57:43.178751 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20014
I20260927 03:57:43.178892 20311 raft_consensus.cc:2229] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:43.179795 20311 raft_consensus.cc:2258] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:43.180497 20311 ts_tablet_manager.cc:1905] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:43.181020 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:43.180855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:41373 (local address 127.0.0.1:46750)
0927 03:57:43.180923 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:43.180976 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 03:57:43.181221 19807 catalog_manager.cc:4584] TS 3d3e534e9b1d4c4c84f690b0a75f0571 (127.1.50.65:41373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abe532c3cff941f7af6b59724fac279d: Network error: Client connection negotiation failed: client connection to 127.1.50.65:41373: connect: Connection refused (error 111)
I20260927 03:57:43.182518 20311 ts_tablet_manager.cc:1918] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:43.182601 20311 log.cc:1196] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/abe532c3cff941f7af6b59724fac279d
I20260927 03:57:43.182850 20311 ts_tablet_manager.cc:1939] T abe532c3cff941f7af6b59724fac279d P a39dabeec4584301ad4668f9fe051a7f: Deleting consensus metadata
I20260927 03:57:43.183338 19810 catalog_manager.cc:4841] TS a39dabeec4584301ad4668f9fe051a7f (127.1.50.67:35261): tablet abe532c3cff941f7af6b59724fac279d (table test-table [id=b16dd7150b2f46538dd36b738ab7f7b1]) successfully deleted
I20260927 03:57:43.184464 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20145
I20260927 03:57:43.189136 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 19792
2026-09-27T03:57:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:43 dist-test-slave-96fg krb5kdc[20316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:43 dist-test-slave-96fg krb5kdc[20316](info): set up 2 sockets
Sep 27 03:57:43 dist-test-slave-96fg krb5kdc[20316](info): commencing operation
krb5kdc: starting...
W20260927 03:57:44.161655 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:57:44.281271 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37737
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:44.363921 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:44.364152 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:44.364183 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:44.365334 20329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:44.365372 20329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:44.365386 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:44.365396 20329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:44.365407 20329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:44.367060 20329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:44.367327 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:44.367575 20329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:44.372695 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.372689 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.372768 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:44.372903 20329 server_base.cc:1034] running on GCE node
I20260927 03:57:44.373258 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:44.373543 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:44.374694 20329 hybrid_clock.cc:648] HybridClock initialized: now 1790481464374679 us; error 29 us; skew 500 ppm
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:44.377251 20329 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:57:44.377699 20329 webserver.cc:460] Webserver started at http://127.1.50.126:41659/ using document root <none> and password file <none>
I20260927 03:57:44.377885 20329 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:44.377928 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:44.378063 20329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:44.379128 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "441af1c9952042faabbf6f799a02ee57"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "e4845ddd603aaf8040c8dd6d41a24885"
server_key_iv: "4d5fab7009612cead7fbebb8f022a1bc"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.379428 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "441af1c9952042faabbf6f799a02ee57"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "e4845ddd603aaf8040c8dd6d41a24885"
server_key_iv: "4d5fab7009612cead7fbebb8f022a1bc"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.380751 20329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:57:44.381537 20344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.381754 20329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:44.381843 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "441af1c9952042faabbf6f799a02ee57"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "e4845ddd603aaf8040c8dd6d41a24885"
server_key_iv: "4d5fab7009612cead7fbebb8f022a1bc"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.381932 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:44.406288 20329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:44.408362 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:44.408483 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:44.412264 20329 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:37737
I20260927 03:57:44.412314 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:37737 every 8 connection(s)
I20260927 03:57:44.412617 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:44.413460 20397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:44.415823 20397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: Bootstrap starting.
I20260927 03:57:44.416363 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20329
I20260927 03:57:44.416399 20397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:44.416458 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:44.416661 1225 external_mini_cluster.cc:1428] Setting key ceae77f74a1085aa6ae2f7476b8862af
I20260927 03:57:44.416663 20397 log.cc:824] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:44.417374 20397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: No bootstrap required, opened a new log
I20260927 03:57:44.419596 20397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } }
I20260927 03:57:44.419745 20397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:44.419780 20397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441af1c9952042faabbf6f799a02ee57, State: Initialized, Role: FOLLOWER
I20260927 03:57:44.419911 20397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } }
I20260927 03:57:44.419994 20397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:44.420037 20397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:44.420087 20397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:44.420697 20397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } }
I20260927 03:57:44.420800 20397 leader_election.cc:302] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441af1c9952042faabbf6f799a02ee57; no voters:
I20260927 03:57:44.421000 20397 leader_election.cc:288] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:44.421056 20402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:44.421209 20402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [term 1 LEADER]: Becoming Leader. State: Replica: 441af1c9952042faabbf6f799a02ee57, State: Running, Role: LEADER
I20260927 03:57:44.421286 20397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:44.421339 20402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } }
I20260927 03:57:44.422817 20403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "441af1c9952042faabbf6f799a02ee57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } } }
I20260927 03:57:44.423086 20403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [sys.catalog]: This master's current role is: LEADER
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:44.422926 20404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441af1c9952042faabbf6f799a02ee57. Latest consensus state: current_term: 1 leader_uuid: "441af1c9952042faabbf6f799a02ee57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441af1c9952042faabbf6f799a02ee57" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 37737 } } }
I20260927 03:57:44.423327 20404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57 [sys.catalog]: This master's current role is: LEADER
W20260927 03:57:44.423399 20417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:44.423480 20417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:44.423538 20419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:44.423694 20419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:44.425297 20419 catalog_manager.cc:1347] Generated new cluster ID: ee39f3d5173641fead9a3e7cdfdf67d8
I20260927 03:57:44.425351 20419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:44.427431 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.417155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.1.50.126:37737)
0927 03:57:44.417325 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.417333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.417357 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:57:44.417812 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.417819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.417865 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.417983 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:44.418604 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.419467 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.420317 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.420721 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.420821 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.423710 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:44.423715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:44.423715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:44.423728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:44.425788 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.426214 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.426217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.426217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.426314 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.426515 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.426516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.426516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.426798 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:44.426979 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.427069 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.427210 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260927 03:57:44.432339 20419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:44.433061 20419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:44.435518 20419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: Generated new TSK 0
I20260927 03:57:44.435708 20419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.50.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
Entry for principal kudu/127.1.50.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab.
I20260927 03:57:44.462180 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:44.543864 20423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:44.544093 20423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:44.544126 20423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:44.545364 20423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:44.545406 20423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:44.545442 20423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:44.546945 20423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.65.keytab
--principal=kudu/127.1.50.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:44.547183 20423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:44.547394 20423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:44.553498 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.553499 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.553498 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:44.553515 20423 server_base.cc:1034] running on GCE node
I20260927 03:57:44.553937 20423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:44.554250 20423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:44.555397 20423 hybrid_clock.cc:648] HybridClock initialized: now 1790481464555383 us; error 32 us; skew 500 ppm
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:44.557971 20423 init.cc:377] Logged in from keytab as kudu/127.1.50.65@KRBTEST.COM (short username kudu)
I20260927 03:57:44.558435 20423 webserver.cc:460] Webserver started at http://127.1.50.65:43915/ using document root <none> and password file <none>
I20260927 03:57:44.558636 20423 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:44.558679 20423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:44.558797 20423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:44.559900 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "bf5f554604754db5b26f27d7143a3d81"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "557e336b7ea19b355b9b7b531c5d22f4"
server_key_iv: "0799cc5f96c0fa5b82d79a7c7e7fcec4"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.560225 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "bf5f554604754db5b26f27d7143a3d81"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "557e336b7ea19b355b9b7b531c5d22f4"
server_key_iv: "0799cc5f96c0fa5b82d79a7c7e7fcec4"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.561514 20423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:57:44.562330 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.562554 20423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:44.562636 20423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "bf5f554604754db5b26f27d7143a3d81"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "557e336b7ea19b355b9b7b531c5d22f4"
server_key_iv: "0799cc5f96c0fa5b82d79a7c7e7fcec4"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.562711 20423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:44.574479 20423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:44.576546 20423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:44.576659 20423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:44.576890 20423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:44.577226 20423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:44.577260 20423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.577280 20423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:44.577302 20423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.582809 20423 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:37903
I20260927 03:57:44.582862 20551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:37903 every 8 connection(s)
I20260927 03:57:44.583160 20423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:44.587208 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20423
I20260927 03:57:44.587301 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:44.587483 1225 external_mini_cluster.cc:1428] Setting key 7f541941548bb11f71b15179367708de
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.65@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:44.593273 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.583848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:50081 (local address 127.1.50.126:37737)
0927 03:57:44.583912 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.583914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.584126 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:57:44.584679 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.584680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.584690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.584751 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:44.585828 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.586460 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.587652 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.587920 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.588004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.590412 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:44.590413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:44.590414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:44.590422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:44.591877 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.592328 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.592329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.592330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.592395 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.592613 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.592614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.592615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.592766 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:44.592838 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.593074 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.593169 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:57:44.593358 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.583834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37737 (local address 127.1.50.65:50081)
0927 03:57:44.584008 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.584049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.584565 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.584803 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.584806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.585462 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:44.585679 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.585684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.586568 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.586569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.587550 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.587554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.588000 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.588001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.588143 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.588727 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:44.588736 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:44.590261 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:44.591980 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.591990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.591993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.592251 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.592460 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.592460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.592460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.592558 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.592826 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:44.592830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:44.592953 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.593033 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.593180 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 03:57:44.593889 20552 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37737
I20260927 03:57:44.593979 20552 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:44.594184 20552 heartbeater.cc:511] Master 127.1.50.126:37737 requested a full tablet report, sending...
I20260927 03:57:44.594834 20361 ts_manager.cc:194] Registered new tserver with Master: bf5f554604754db5b26f27d7143a3d81 (127.1.50.65:37903)
I20260927 03:57:44.595762 20361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.65@KRBTEST.COM'} at 127.1.50.65:50081
WARNING: no policy specified for kudu/127.1.50.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
Entry for principal kudu/127.1.50.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab.
I20260927 03:57:44.615330 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:44.697723 20557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:44.697947 20557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:44.697973 20557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:44.699213 20557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:44.699261 20557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:44.699288 20557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:44.700758 20557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.66.keytab
--principal=kudu/127.1.50.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:44.700989 20557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:44.701215 20557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:44.706246 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.706233 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.706370 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:44.706964 20557 server_base.cc:1034] running on GCE node
I20260927 03:57:44.707206 20557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:44.707414 20557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:44.708576 20557 hybrid_clock.cc:648] HybridClock initialized: now 1790481464708557 us; error 29 us; skew 500 ppm
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:44.711076 20557 init.cc:377] Logged in from keytab as kudu/127.1.50.66@KRBTEST.COM (short username kudu)
I20260927 03:57:44.711499 20557 webserver.cc:460] Webserver started at http://127.1.50.66:39327/ using document root <none> and password file <none>
I20260927 03:57:44.711678 20557 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:44.711716 20557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:44.711807 20557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:44.712795 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "95516aa4190441569929fb21746417eb"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "f4173a1c649c4f41bea93b9080befbef"
server_key_iv: "1c172251e1336e32be919973a5a41ac2"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.713081 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "95516aa4190441569929fb21746417eb"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "f4173a1c649c4f41bea93b9080befbef"
server_key_iv: "1c172251e1336e32be919973a5a41ac2"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.714290 20557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:44.714989 20572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.715188 20557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:44.715261 20557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "95516aa4190441569929fb21746417eb"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "f4173a1c649c4f41bea93b9080befbef"
server_key_iv: "1c172251e1336e32be919973a5a41ac2"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.715314 20557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:44.733497 20557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:44.735781 20557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:44.735906 20557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:44.736117 20557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:44.736367 20557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:44.736397 20557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.736418 20557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:44.736428 20557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.742269 20557 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:42381
I20260927 03:57:44.742316 20685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:42381 every 8 connection(s)
I20260927 03:57:44.742635 20557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.66@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:44.750557 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20557
I20260927 03:57:44.750655 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:44.750859 1225 external_mini_cluster.cc:1428] Setting key de3d10364eb6656b948311baaa94d1c5
I20260927 03:57:44.752584 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.743382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:45571 (local address 127.1.50.126:37737)
0927 03:57:44.743473 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.743476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.743647 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:57:44.744202 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.744203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.744217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.744272 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:44.745302 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.745823 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.746865 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.747072 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.747138 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.749495 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:44.749497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:44.749497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:44.749506 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:44.751078 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.751611 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.751612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.751612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.751692 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.751920 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.751921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.751922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.752038 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:44.752095 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.752407 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.752493 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:57:44.752784 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.743307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37737 (local address 127.1.50.66:45571)
0927 03:57:44.743515 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.743556 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.744090 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.744357 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.744361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.744991 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:44.745195 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.745201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.745918 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.745919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.746767 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.746769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.747170 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.747171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.747292 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.747843 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:44.747851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:44.749386 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:44.751233 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.751243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.751246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.751538 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.751753 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.751753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.751754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.751863 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.752092 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:44.752100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:44.752281 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.752373 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.752554 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260927 03:57:44.753360 20686 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37737
I20260927 03:57:44.753463 20686 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:44.753655 20686 heartbeater.cc:511] Master 127.1.50.126:37737 requested a full tablet report, sending...
I20260927 03:57:44.754081 20361 ts_manager.cc:194] Registered new tserver with Master: 95516aa4190441569929fb21746417eb (127.1.50.66:42381)
I20260927 03:57:44.754454 20361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.66@KRBTEST.COM'} at 127.1.50.66:45571
WARNING: no policy specified for kudu/127.1.50.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
Entry for principal kudu/127.1.50.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab.
I20260927 03:57:44.777289 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:44.857929 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:44.858173 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:44.858196 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:44.859396 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:44.859447 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:44.859474 20691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:44.860978 20691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.67.keytab
--principal=kudu/127.1.50.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:37737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:44.861225 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:44.861452 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:44.866722 20696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.866823 20699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:44.866722 20697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:44.867038 20691 server_base.cc:1034] running on GCE node
I20260927 03:57:44.867290 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:44.867544 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:44.868717 20691 hybrid_clock.cc:648] HybridClock initialized: now 1790481464868699 us; error 36 us; skew 500 ppm
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:44.871261 20691 init.cc:377] Logged in from keytab as kudu/127.1.50.67@KRBTEST.COM (short username kudu)
I20260927 03:57:44.871665 20691 webserver.cc:460] Webserver started at http://127.1.50.67:35153/ using document root <none> and password file <none>
I20260927 03:57:44.871891 20691 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:44.871935 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:44.872049 20691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:44.873044 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "b95cdffc3a3b4e4cb2322c9324749ec6"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "9b6ea1fc46886f4fdb052e80177f2649"
server_key_iv: "1fe5fc8bd995072b487518b1a1492773"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.873337 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "b95cdffc3a3b4e4cb2322c9324749ec6"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "9b6ea1fc46886f4fdb052e80177f2649"
server_key_iv: "1fe5fc8bd995072b487518b1a1492773"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.874708 20691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:44.875483 20706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.875692 20691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:44.875768 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "b95cdffc3a3b4e4cb2322c9324749ec6"
format_stamp: "Formatted at 2026-09-27 03:57:44 on dist-test-slave-96fg"
server_key: "9b6ea1fc46886f4fdb052e80177f2649"
server_key_iv: "1fe5fc8bd995072b487518b1a1492773"
server_key_version: "encryptionkey@0"
I20260927 03:57:44.875833 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:44.887208 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:44.889106 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:44.889227 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:44.889438 20691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:44.889725 20691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:44.889755 20691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.889777 20691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:44.889801 20691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:44.895414 20691 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:41205
I20260927 03:57:44.895493 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:41205 every 8 connection(s)
I20260927 03:57:44.895771 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:57:44 dist-test-slave-96fg krb5kdc[20316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.67@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:57:44.902196 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20691
I20260927 03:57:44.902307 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:44.902469 1225 external_mini_cluster.cc:1428] Setting key b1448bd66ca24565f12f04aa3d550c63
I20260927 03:57:44.904947 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.896571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:55345 (local address 127.1.50.126:37737)
0927 03:57:44.896614 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.896615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.896806 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:57:44.897302 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.897303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.897316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.897362 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:44.898412 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.898919 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.899871 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.900063 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.900116 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.902287 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:44.902288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:44.902289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:44.902296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:44.903741 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.904086 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.904087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.904088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.904142 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.904365 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.904366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.904366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.904477 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:44.904530 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.904781 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.904857 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:57:44.905138 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.896529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:37737 (local address 127.1.50.67:55345)
0927 03:57:44.896678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.896722 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.897211 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.897437 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.897441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.898004 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:44.898317 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.898323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.899002 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.899002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.899793 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.899795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.900137 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.900138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.900250 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.900740 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:57:44.900748 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:44.902204 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:44.903809 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.903816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.903818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.904025 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.904207 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:57:44.904207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:57:44.904208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:57:44.904311 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:57:44.904529 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:44.904535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:44.904670 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.904756 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.904912 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 03:57:44.905591 20820 heartbeater.cc:347] Connected to a master server at 127.1.50.126:37737
I20260927 03:57:44.905689 20820 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:44.905833 20820 heartbeater.cc:511] Master 127.1.50.126:37737 requested a full tablet report, sending...
I20260927 03:57:44.906229 20361 ts_manager.cc:194] Registered new tserver with Master: b95cdffc3a3b4e4cb2322c9324749ec6 (127.1.50.67:41205)
I20260927 03:57:44.906576 20361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.50.67@KRBTEST.COM'} at 127.1.50.67:55345
I20260927 03:57:44.914346 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:44.921478 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.915620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.1.50.126:37737)
0927 03:57:44.915694 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.915696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.915793 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:57:44.915837 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.915838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.915847 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.915894 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:44.916514 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.917015 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.917703 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.917917 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.917973 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.919221 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:44.919222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:44.919222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:44.919228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:44.920308 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.920647 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.920648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.920648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.920706 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:44.920866 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:44.920866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:44.920867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:44.921086 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:44.921174 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.921324 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.921396 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:57:44.923249 20361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:44.924719 20361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:44.931037 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.926924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.1.50.66:42381)
0927 03:57:44.927298 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.927303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.927314 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:44.927353 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.927355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.927371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.927461 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:44.927592 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.928494 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.929847 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930605 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.930738 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.930905 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.930950 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":73,"threads_started":1}
I20260927 03:57:44.931041 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.926798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:42381 (local address 127.0.0.1:57448)
0927 03:57:44.927055 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.927100 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.927200 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.927464 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.927467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.927470 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:44.927532 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.927534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.928582 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.928583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.929751 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.929754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930696 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.930790 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.930805 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.930879 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.930936 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260927 03:57:44.931041 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.927062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:37903 (local address 127.0.0.1:35042)
0927 03:57:44.927670 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.927680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.927726 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.927963 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.927964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.927966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:44.928008 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.928009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.929178 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.929179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.930082 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.930085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930790 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.930855 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.930858 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.930900 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.930936 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260927 03:57:44.931202 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.927241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.1.50.65:37903)
0927 03:57:44.927405 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.927410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.927776 (+ 366us) server_negotiation.cc:407] Connection header received
0927 03:57:44.927816 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.927818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.927831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.927890 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:44.928086 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.929086 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930149 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930625 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930700 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.930760 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.931082 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.931125 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260927 03:57:44.931478 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.926950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:41205 (local address 127.0.0.1:46162)
0927 03:57:44.927068 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:44.927100 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:44.927168 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:44.927578 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:44.927579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:44.927581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:44.927685 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.927687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.928970 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.928971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.930088 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:44.930091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930994 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:44.931064 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.931068 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:44.931369 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 03:57:44.931411 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":34,"thread_start_us":37,"threads_started":1}
I20260927 03:57:44.931876 20486 tablet_service.cc:1467] Processing CreateTablet for tablet ce6d6351503948338647418632014b9b (DEFAULT_TABLE table=test-table [id=22250e54fc2d46cf8eaafb66320c494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:44.932250 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6d6351503948338647418632014b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:44.932160 20620 tablet_service.cc:1467] Processing CreateTablet for tablet ce6d6351503948338647418632014b9b (DEFAULT_TABLE table=test-table [id=22250e54fc2d46cf8eaafb66320c494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:44.932461 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6d6351503948338647418632014b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:44.934888 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:44.927073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.1.50.67:41205)
0927 03:57:44.927426 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:44.927429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:44.927438 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:44.927473 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:44.927475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:44.927488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:44.927572 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:44.927766 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.928862 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930173 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:44.930853 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:44.930913 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:44.930960 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:44.931403 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 03:57:44.934804 (+ 3401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":83,"threads_started":1}
I20260927 03:57:44.934888 20834 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: Bootstrap starting.
I20260927 03:57:44.935228 20835 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Bootstrap starting.
I20260927 03:57:44.935542 20834 tablet_bootstrap.cc:654] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:44.935568 20754 tablet_service.cc:1467] Processing CreateTablet for tablet ce6d6351503948338647418632014b9b (DEFAULT_TABLE table=test-table [id=22250e54fc2d46cf8eaafb66320c494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:44.935787 20835 tablet_bootstrap.cc:654] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:44.935906 20754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6d6351503948338647418632014b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:44.936151 20834 log.cc:824] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:44.936311 20835 log.cc:824] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:44.936925 20834 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: No bootstrap required, opened a new log
I20260927 03:57:44.936993 20834 ts_tablet_manager.cc:1397] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:44.938409 20835 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: No bootstrap required, opened a new log
I20260927 03:57:44.938470 20835 ts_tablet_manager.cc:1397] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:57:44.938930 20834 raft_consensus.cc:348] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.939137 20834 raft_consensus.cc:374] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:44.939182 20834 raft_consensus.cc:729] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f554604754db5b26f27d7143a3d81, State: Initialized, Role: FOLLOWER
I20260927 03:57:44.939298 20838 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Bootstrap starting.
I20260927 03:57:44.939296 20834 consensus_queue.cc:260] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.939566 20834 ts_tablet_manager.cc:1428] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:44.939611 20552 heartbeater.cc:503] Master 127.1.50.126:37737 was elected leader, sending a full tablet report...
I20260927 03:57:44.939989 20838 tablet_bootstrap.cc:654] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:44.940569 20838 log.cc:824] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:44.940582 20835 raft_consensus.cc:348] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.940760 20835 raft_consensus.cc:374] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:44.940804 20835 raft_consensus.cc:729] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95516aa4190441569929fb21746417eb, State: Initialized, Role: FOLLOWER
I20260927 03:57:44.940893 20835 consensus_queue.cc:260] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.941078 20835 ts_tablet_manager.cc:1428] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:44.941133 20686 heartbeater.cc:503] Master 127.1.50.126:37737 was elected leader, sending a full tablet report...
I20260927 03:57:44.941684 20838 tablet_bootstrap.cc:492] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: No bootstrap required, opened a new log
I20260927 03:57:44.941753 20838 ts_tablet_manager.cc:1397] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:57:44.943185 20838 raft_consensus.cc:348] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.943318 20838 raft_consensus.cc:374] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:44.943338 20838 raft_consensus.cc:729] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95cdffc3a3b4e4cb2322c9324749ec6, State: Initialized, Role: FOLLOWER
I20260927 03:57:44.943419 20838 consensus_queue.cc:260] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:44.943646 20838 ts_tablet_manager.cc:1428] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:57:44.943694 20820 heartbeater.cc:503] Master 127.1.50.126:37737 was elected leader, sending a full tablet report...
W20260927 03:57:44.993228 20687 tablet.cc:2369] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:45.000036 20842 raft_consensus.cc:482] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:45.000169 20842 raft_consensus.cc:504] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:45.000524 20842 leader_election.cc:288] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95516aa4190441569929fb21746417eb (127.1.50.66:42381), bf5f554604754db5b26f27d7143a3d81 (127.1.50.65:37903)
I20260927 03:57:45.003533 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.000827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:44527 (local address 127.1.50.66:42381)
0927 03:57:45.000874 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.000876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.000888 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:57:45.000918 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.000919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.000929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.000974 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.001198 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.001842 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.002833 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.003216 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.003277 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.003287 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.003433 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.003468 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:57:45.003549 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.000931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:47045 (local address 127.1.50.65:37903)
0927 03:57:45.000981 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.000984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.001053 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:57:45.001083 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.001084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.001096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.001147 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.001300 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.001883 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.002840 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.003214 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.003274 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.003285 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.003433 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.003478 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:57:45.003549 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.000727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:42381 (local address 127.1.50.67:44527)
0927 03:57:45.000781 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.000796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.000864 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.000998 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.000999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.001001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.001069 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.001072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.001917 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.001918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.002745 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.002747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.003290 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.003291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.003370 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.003372 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.003419 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.003462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260927 03:57:45.003557 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.000835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:37903 (local address 127.1.50.67:47045)
0927 03:57:45.000992 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.001001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.001047 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.001164 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.001164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.001166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.001243 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.001245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.001954 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.001955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.002748 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.002750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.003288 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.003289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.003365 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.003369 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.003416 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.003457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260927 03:57:45.003912 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6d6351503948338647418632014b9b" candidate_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95516aa4190441569929fb21746417eb" is_pre_election: true
I20260927 03:57:45.004055 20640 raft_consensus.cc:2454] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95cdffc3a3b4e4cb2322c9324749ec6 in term 0.
I20260927 03:57:45.004289 20710 leader_election.cc:302] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95516aa4190441569929fb21746417eb, b95cdffc3a3b4e4cb2322c9324749ec6; no voters:
I20260927 03:57:45.004403 20506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6d6351503948338647418632014b9b" candidate_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f554604754db5b26f27d7143a3d81" is_pre_election: true
I20260927 03:57:45.004427 20842 raft_consensus.cc:2781] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:45.004488 20842 raft_consensus.cc:482] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:45.004513 20842 raft_consensus.cc:3037] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.004520 20506 raft_consensus.cc:2454] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95cdffc3a3b4e4cb2322c9324749ec6 in term 0.
I20260927 03:57:45.005234 20842 raft_consensus.cc:504] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:45.005347 20842 leader_election.cc:288] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 95516aa4190441569929fb21746417eb (127.1.50.66:42381), bf5f554604754db5b26f27d7143a3d81 (127.1.50.65:37903)
I20260927 03:57:45.005668 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6d6351503948338647418632014b9b" candidate_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95516aa4190441569929fb21746417eb"
I20260927 03:57:45.005661 20506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6d6351503948338647418632014b9b" candidate_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f554604754db5b26f27d7143a3d81"
I20260927 03:57:45.005757 20506 raft_consensus.cc:3037] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.005757 20640 raft_consensus.cc:3037] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.006668 20640 raft_consensus.cc:2454] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95cdffc3a3b4e4cb2322c9324749ec6 in term 1.
I20260927 03:57:45.006693 20506 raft_consensus.cc:2454] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95cdffc3a3b4e4cb2322c9324749ec6 in term 1.
I20260927 03:57:45.006861 20710 leader_election.cc:302] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95516aa4190441569929fb21746417eb, b95cdffc3a3b4e4cb2322c9324749ec6; no voters:
I20260927 03:57:45.006937 20842 raft_consensus.cc:2781] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:45.007133 20842 raft_consensus.cc:686] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 1 LEADER]: Becoming Leader. State: Replica: b95cdffc3a3b4e4cb2322c9324749ec6, State: Running, Role: LEADER
I20260927 03:57:45.007220 20842 consensus_queue.cc:237] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } }
I20260927 03:57:45.008073 20360 catalog_manager.cc:5495] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b95cdffc3a3b4e4cb2322c9324749ec6 (127.1.50.67). New cstate: current_term: 1 leader_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b95cdffc3a3b4e4cb2322c9324749ec6" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 41205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 } health_report { overall_health: UNKNOWN } } }
I20260927 03:57:45.024907 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.022666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.1.50.67:41205)
0927 03:57:45.022711 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.022713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.022721 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:45.022759 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.022760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.022778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.022833 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:45.022940 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.023409 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.024224 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.024430 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.024489 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.024624 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:45.024730 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:45.024783 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.024799 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.024824 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:57:45.026711 20506 raft_consensus.cc:1261] T ce6d6351503948338647418632014b9b P bf5f554604754db5b26f27d7143a3d81 [term 1 FOLLOWER]: Refusing update from remote peer b95cdffc3a3b4e4cb2322c9324749ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:45.026739 20640 raft_consensus.cc:1261] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 1 FOLLOWER]: Refusing update from remote peer b95cdffc3a3b4e4cb2322c9324749ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:45.027047 20842 consensus_queue.cc:1035] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5f554604754db5b26f27d7143a3d81" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:45.027186 20844 consensus_queue.cc:1035] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95516aa4190441569929fb21746417eb" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:45.032066 20361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:45.032162 20361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:45.032855 20361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441af1c9952042faabbf6f799a02ee57: Sending DeleteTablet for 3 replicas of tablet ce6d6351503948338647418632014b9b
I20260927 03:57:45.033221 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6d6351503948338647418632014b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:57448
I20260927 03:57:45.033244 20754 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6d6351503948338647418632014b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:46162
I20260927 03:57:45.033365 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20423
I20260927 03:57:45.033434 20486 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6d6351503948338647418632014b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='kudu', principal='kudu/127.1.50.126@KRBTEST.COM'} at 127.0.0.1:35042
I20260927 03:57:45.033535 20857 tablet_replica.cc:330] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: stopping tablet replica
I20260927 03:57:45.033638 20857 raft_consensus.cc:2229] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:45.033731 20857 raft_consensus.cc:2258] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:45.034160 20858 tablet_replica.cc:330] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: stopping tablet replica
I20260927 03:57:45.034235 20857 ts_tablet_manager.cc:1905] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:45.035974 20857 ts_tablet_manager.cc:1918] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:45.036038 20857 log.cc:1196] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/ce6d6351503948338647418632014b9b
I20260927 03:57:45.036350 20857 ts_tablet_manager.cc:1939] T ce6d6351503948338647418632014b9b P 95516aa4190441569929fb21746417eb: Deleting consensus metadata
I20260927 03:57:45.036801 20348 catalog_manager.cc:4841] TS 95516aa4190441569929fb21746417eb (127.1.50.66:42381): tablet ce6d6351503948338647418632014b9b (table test-table [id=22250e54fc2d46cf8eaafb66320c494a]) successfully deleted
I20260927 03:57:45.037809 20858 raft_consensus.cc:2229] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:57:45.037974 20858 raft_consensus.cc:2258] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:45.038465 20858 ts_tablet_manager.cc:1905] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:45.039778 20858 ts_tablet_manager.cc:1918] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:45.039844 20858 log.cc:1196] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/ce6d6351503948338647418632014b9b
I20260927 03:57:45.040064 20858 ts_tablet_manager.cc:1939] T ce6d6351503948338647418632014b9b P b95cdffc3a3b4e4cb2322c9324749ec6: Deleting consensus metadata
I20260927 03:57:45.040580 20348 catalog_manager.cc:4841] TS b95cdffc3a3b4e4cb2322c9324749ec6 (127.1.50.67:41205): tablet ce6d6351503948338647418632014b9b (table test-table [id=22250e54fc2d46cf8eaafb66320c494a]) successfully deleted
W20260927 03:57:45.041116 20348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:37903 (error 108)
I20260927 03:57:45.041594 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.041479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:37903 (local address 127.0.0.1:35052)
0927 03:57:45.041524 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.041561 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:37903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:57:45.041666 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20557
W20260927 03:57:45.041742 20348 catalog_manager.cc:4584] TS bf5f554604754db5b26f27d7143a3d81 (127.1.50.65:37903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce6d6351503948338647418632014b9b: Network error: Client connection negotiation failed: client connection to 127.1.50.65:37903: connect: Connection refused (error 111)
I20260927 03:57:45.046172 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20691
I20260927 03:57:45.050311 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20329
2026-09-27T03:57:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T03:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:45Z Disabled control of system clock
I20260927 03:57:45.076781 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:45739
--webserver_interface=127.1.50.126
--webserver_port=0
--builtin_ntp_servers=127.1.50.84:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:45.156847 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:45.157079 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:45.157100 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:45.158248 20863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:45.158283 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:45.158294 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:45.158305 20863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:45.158315 20863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:45.159590 20863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:45739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:45.159786 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:45.160036 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:45.164844 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.164870 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.164847 20863 server_base.cc:1034] running on GCE node
W20260927 03:57:45.164875 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.165297 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:45.165565 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:45.166735 20863 hybrid_clock.cc:648] HybridClock initialized: now 1790481465166715 us; error 34 us; skew 500 ppm
I20260927 03:57:45.168121 20863 webserver.cc:460] Webserver started at http://127.1.50.126:37509/ using document root <none> and password file <none>
I20260927 03:57:45.168375 20863 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:45.168419 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:45.168526 20863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:45.169611 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "7f1adbe5503b4584bdbb2bf6e075a02b"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "a06d5e4234beaf67bc6b60995b277b75"
server_key_iv: "f564ed05eba18ef70a77e2741959eb85"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.169932 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "7f1adbe5503b4584bdbb2bf6e075a02b"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "a06d5e4234beaf67bc6b60995b277b75"
server_key_iv: "f564ed05eba18ef70a77e2741959eb85"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.171257 20863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:45.172003 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.172237 20863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:45.172333 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "7f1adbe5503b4584bdbb2bf6e075a02b"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "a06d5e4234beaf67bc6b60995b277b75"
server_key_iv: "f564ed05eba18ef70a77e2741959eb85"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.172401 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:45.187933 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:45.188253 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:45.188390 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:45.192416 20863 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:45739
I20260927 03:57:45.192502 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:45739 every 8 connection(s)
I20260927 03:57:45.192807 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:45.193770 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:45.196069 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: Bootstrap starting.
I20260927 03:57:45.196637 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:45.196910 20930 log.cc:824] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:45.197660 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: No bootstrap required, opened a new log
I20260927 03:57:45.199347 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } }
I20260927 03:57:45.199488 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:45.199509 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1adbe5503b4584bdbb2bf6e075a02b, State: Initialized, Role: FOLLOWER
I20260927 03:57:45.199642 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } }
I20260927 03:57:45.199710 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:45.199733 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:45.199777 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.200309 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } }
I20260927 03:57:45.200392 20930 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1adbe5503b4584bdbb2bf6e075a02b; no voters:
I20260927 03:57:45.200559 20930 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:45.200629 20933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:45.200827 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:45.200827 20933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [term 1 LEADER]: Becoming Leader. State: Replica: 7f1adbe5503b4584bdbb2bf6e075a02b, State: Running, Role: LEADER
I20260927 03:57:45.201000 20933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } }
I20260927 03:57:45.201256 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20863
I20260927 03:57:45.201360 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:45.201493 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1adbe5503b4584bdbb2bf6e075a02b. Latest consensus state: current_term: 1 leader_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } } }
I20260927 03:57:45.201547 1225 external_mini_cluster.cc:1428] Setting key 8a4774681e94854d96414ab3710d515f
I20260927 03:57:45.201663 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:45.201607 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1adbe5503b4584bdbb2bf6e075a02b" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 45739 } } }
I20260927 03:57:45.202363 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b [sys.catalog]: This master's current role is: LEADER
W20260927 03:57:45.202659 20950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:45.202714 20950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:45.202942 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:45.203143 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:45.204401 20951 catalog_manager.cc:1347] Generated new cluster ID: d215fa69d9494fc681ecfdc950bc3f1d
I20260927 03:57:45.204452 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:45.207562 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.202062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.1.50.126:45739)
0927 03:57:45.202288 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.202297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.202406 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:57:45.202657 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.202665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.202727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.202857 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:45.203087 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.204389 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.205478 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.206129 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.206249 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.206859 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:45.206863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:45.206863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:45.207083 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:45.207126 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:45.207141 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:45.207207 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.207261 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.207337 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":57,"threads_started":1}
I20260927 03:57:45.212252 20951 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:45.212714 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:45.219537 20951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: Generated new TSK 0
I20260927 03:57:45.219812 20951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:45.229775 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.65:0
--local_ip_for_outbound_sockets=127.1.50.65
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--builtin_ntp_servers=127.1.50.84:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:57:45.311547 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:45.311771 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:45.311794 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:45.312989 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:45.313033 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:45.313058 20954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.65
I20260927 03:57:45.314532 20954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.50.65
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.65
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:45.314774 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:45.315127 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:45.320892 20960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.320994 20962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.320925 20959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.321334 20954 server_base.cc:1034] running on GCE node
I20260927 03:57:45.321509 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:45.321748 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:45.322897 20954 hybrid_clock.cc:648] HybridClock initialized: now 1790481465322884 us; error 26 us; skew 500 ppm
I20260927 03:57:45.324208 20954 webserver.cc:460] Webserver started at http://127.1.50.65:46353/ using document root <none> and password file <none>
I20260927 03:57:45.324430 20954 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:45.324477 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:45.324604 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:45.325701 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data/instance:
uuid: "94166f1e172d464ebc1153c020ce7dd7"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "dd53233a20fa1579e711efc1ba6b6aab"
server_key_iv: "1ba8f506a7f948763298bd95f666a40a"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.326076 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance:
uuid: "94166f1e172d464ebc1153c020ce7dd7"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "dd53233a20fa1579e711efc1ba6b6aab"
server_key_iv: "1ba8f506a7f948763298bd95f666a40a"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.327307 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:45.328024 20968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.328217 20954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:57:45.328321 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal
uuid: "94166f1e172d464ebc1153c020ce7dd7"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "dd53233a20fa1579e711efc1ba6b6aab"
server_key_iv: "1ba8f506a7f948763298bd95f666a40a"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.328420 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:45.337355 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:45.337620 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:45.337718 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:45.337949 20954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:45.338289 20954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:45.338322 20954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.338343 20954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:45.338363 20954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.343684 20954 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.65:34987
I20260927 03:57:45.343725 21081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.65:34987 every 8 connection(s)
I20260927 03:57:45.344041 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/data/info.pb
I20260927 03:57:45.344585 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 20954
I20260927 03:57:45.344687 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-0/wal/instance
I20260927 03:57:45.344882 1225 external_mini_cluster.cc:1428] Setting key f77909100ad03f53cd3bc5eb90414081
I20260927 03:57:45.345780 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.66:0
--local_ip_for_outbound_sockets=127.1.50.66
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--builtin_ntp_servers=127.1.50.84:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:57:45.350069 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.344740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:55107 (local address 127.1.50.126:45739)
0927 03:57:45.344786 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.344789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.345128 (+ 339us) server_negotiation.cc:407] Connection header received
0927 03:57:45.345590 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.345591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.345603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.345673 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:45.346362 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.347050 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.348438 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.348652 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.348714 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.349693 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:45.349694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:45.349694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:45.349699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:45.349721 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:45.349726 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:45.349774 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.349902 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.349941 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:57:45.350271 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.344734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45739 (local address 127.1.50.65:55107)
0927 03:57:45.344915 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.344957 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.345510 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.345723 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.345726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.345734 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:45.346284 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.346289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.347171 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.347173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.348343 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.348346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.348740 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.348741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.348894 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.349504 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:45.349504 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:45.349547 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:45.349581 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:45.349790 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:45.349796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:45.349799 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.349873 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.349996 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260927 03:57:45.350769 21082 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45739
I20260927 03:57:45.350862 21082 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:45.351047 21082 heartbeater.cc:511] Master 127.1.50.126:45739 requested a full tablet report, sending...
I20260927 03:57:45.351543 20894 ts_manager.cc:194] Registered new tserver with Master: 94166f1e172d464ebc1153c020ce7dd7 (127.1.50.65:34987)
I20260927 03:57:45.352190 20894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.65:55107
W20260927 03:57:45.426077 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:45.426291 21085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:45.426321 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:45.427596 21085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:45.427646 21085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:45.427681 21085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.66
I20260927 03:57:45.429061 21085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.50.66
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.66
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:45.429316 21085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:45.429610 21085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:45.434675 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.434736 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.434788 21085 server_base.cc:1034] running on GCE node
W20260927 03:57:45.434661 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.435055 21085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:45.435354 21085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:45.436517 21085 hybrid_clock.cc:648] HybridClock initialized: now 1790481465436501 us; error 33 us; skew 500 ppm
I20260927 03:57:45.437753 21085 webserver.cc:460] Webserver started at http://127.1.50.66:36245/ using document root <none> and password file <none>
I20260927 03:57:45.437981 21085 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:45.438161 21085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:45.438299 21085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:45.439342 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data/instance:
uuid: "45c82e12f07b4b329e7a1aa19506d443"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "d2bbd746108801b3e55545ee55ed648d"
server_key_iv: "4b724a72c78d61471de2f45a314d16f4"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.439647 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance:
uuid: "45c82e12f07b4b329e7a1aa19506d443"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "d2bbd746108801b3e55545ee55ed648d"
server_key_iv: "4b724a72c78d61471de2f45a314d16f4"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.440898 21085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:57:45.441702 21099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.441859 21085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:45.441942 21085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal
uuid: "45c82e12f07b4b329e7a1aa19506d443"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "d2bbd746108801b3e55545ee55ed648d"
server_key_iv: "4b724a72c78d61471de2f45a314d16f4"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.442066 21085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:45.459625 21085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:45.459933 21085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:45.460068 21085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:45.460291 21085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:45.460618 21085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:45.460659 21085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.460688 21085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:45.460711 21085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.466300 21085 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.66:41211
I20260927 03:57:45.466396 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.66:41211 every 8 connection(s)
I20260927 03:57:45.466678 21085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/data/info.pb
I20260927 03:57:45.471278 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21085
I20260927 03:57:45.471369 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal/instance
I20260927 03:57:45.471542 1225 external_mini_cluster.cc:1428] Setting key f891fd6c3aa22b99cf7f6fc47fc74ea7
I20260927 03:57:45.472052 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.467357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.66:48807 (local address 127.1.50.126:45739)
0927 03:57:45.467402 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.467405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.467633 (+ 228us) server_negotiation.cc:407] Connection header received
0927 03:57:45.468145 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.468146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.468158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.468218 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:45.469011 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.469524 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.470564 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.470769 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.470824 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.471736 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:45.471738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:45.471739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:45.471744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:45.471762 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:45.471766 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:45.471867 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.471941 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.471969 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:57:45.472226 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.467336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45739 (local address 127.1.50.66:48807)
0927 03:57:45.467498 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.467551 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.468042 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.468287 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.468291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.468307 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:45.468900 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.468906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.469643 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.469644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.470485 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.470488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.470839 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.470840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.470961 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.471583 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:45.471584 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:45.471624 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:45.471654 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:45.471866 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:45.471872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:45.471875 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.471942 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.472025 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260927 03:57:45.472618 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.50.67:0
--local_ip_for_outbound_sockets=127.1.50.67
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--builtin_ntp_servers=127.1.50.84:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:57:45.472715 21213 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45739
I20260927 03:57:45.472815 21213 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:45.473004 21213 heartbeater.cc:511] Master 127.1.50.126:45739 requested a full tablet report, sending...
I20260927 03:57:45.473366 20894 ts_manager.cc:194] Registered new tserver with Master: 45c82e12f07b4b329e7a1aa19506d443 (127.1.50.66:41211)
I20260927 03:57:45.473825 20894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.66:48807
W20260927 03:57:45.551628 21216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:45.551847 21216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:45.551877 21216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:45.553068 21216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:45.553110 21216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:45.553143 21216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.50.67
I20260927 03:57:45.554591 21216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.50.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.50.67
--webserver_port=0
--tserver_master_addrs=127.1.50.126:45739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.50.67
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:45.554852 21216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:45.555145 21216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:45.560297 21221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.560374 21222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:45.560410 21224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:45.560413 21216 server_base.cc:1034] running on GCE node
I20260927 03:57:45.560783 21216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:45.561004 21216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:45.562143 21216 hybrid_clock.cc:648] HybridClock initialized: now 1790481465562108 us; error 46 us; skew 500 ppm
I20260927 03:57:45.563517 21216 webserver.cc:460] Webserver started at http://127.1.50.67:36245/ using document root <none> and password file <none>
I20260927 03:57:45.563762 21216 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:45.563825 21216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:45.563938 21216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:45.565094 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data/instance:
uuid: "f4dc4a1ad9e14731a791c987bd9443ec"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "56cb531cf85ab397ff17fdd639871286"
server_key_iv: "1bb0b838a0c0ff2adbd9c12a9b02c347"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.565416 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance:
uuid: "f4dc4a1ad9e14731a791c987bd9443ec"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "56cb531cf85ab397ff17fdd639871286"
server_key_iv: "1bb0b838a0c0ff2adbd9c12a9b02c347"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.566773 21216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:45.567633 21230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.567926 21216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:45.568022 21216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal
uuid: "f4dc4a1ad9e14731a791c987bd9443ec"
format_stamp: "Formatted at 2026-09-27 03:57:45 on dist-test-slave-96fg"
server_key: "56cb531cf85ab397ff17fdd639871286"
server_key_iv: "1bb0b838a0c0ff2adbd9c12a9b02c347"
server_key_version: "encryptionkey@0"
I20260927 03:57:45.568089 21216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:45.597807 21216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:45.598300 21216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:45.598476 21216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:45.598733 21216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:57:45.599074 21216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:57:45.599110 21216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.599140 21216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:57:45.599156 21216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:45.604646 21216 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.67:42633
I20260927 03:57:45.604694 21343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.67:42633 every 8 connection(s)
I20260927 03:57:45.605018 21216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/data/info.pb
I20260927 03:57:45.607951 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21216
I20260927 03:57:45.608062 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal/instance
I20260927 03:57:45.608218 1225 external_mini_cluster.cc:1428] Setting key 7ce17936d27099bdd53dd7fc13ad38ac
I20260927 03:57:45.610284 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.605703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.67:37215 (local address 127.1.50.126:45739)
0927 03:57:45.605751 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.605753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.605982 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:57:45.606514 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.606516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.606529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.606578 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:45.607267 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.607770 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.608971 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.609226 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.609283 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.610058 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:45.610059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:45.610060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:45.610065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:45.610081 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:45.610085 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:45.610136 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.610201 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.610227 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:57:45.610476 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.605681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:45739 (local address 127.1.50.67:37215)
0927 03:57:45.605866 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.605898 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.606416 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.606647 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.606651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.606661 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:57:45.607178 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.607183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.607857 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.607858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.608893 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.608896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.609294 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.609295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.609404 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.609904 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:57:45.609904 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:57:45.609933 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:57:45.609956 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:57:45.610143 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:57:45.610147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:57:45.610149 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.610198 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.610276 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260927 03:57:45.610882 21344 heartbeater.cc:347] Connected to a master server at 127.1.50.126:45739
I20260927 03:57:45.610966 21344 heartbeater.cc:464] Registering TS with master...
I20260927 03:57:45.611112 21344 heartbeater.cc:511] Master 127.1.50.126:45739 requested a full tablet report, sending...
I20260927 03:57:45.611466 20894 ts_manager.cc:194] Registered new tserver with Master: f4dc4a1ad9e14731a791c987bd9443ec (127.1.50.67:42633)
I20260927 03:57:45.611824 20894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.50.67:37215
I20260927 03:57:45.619864 1225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:57:45.623934 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.621195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.1.50.126:45739)
0927 03:57:45.621263 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.621265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.621362 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:57:45.621407 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.621408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.621420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.621486 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:45.621650 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.622217 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.622916 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.623096 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.623149 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.623721 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:45.623722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:45.623723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:45.623727 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:45.623740 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:45.623744 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:45.623782 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.623827 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.623850 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:57:45.625514 20894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:57:45.627223 20894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:57:45.633881 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.1.50.67:42633)
0927 03:57:45.629913 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.629917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.629926 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:57:45.629959 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.629961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.629974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.630218 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.630279 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631433 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.632650 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633454 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633626 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.633750 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.633795 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":79,"threads_started":1}
I20260927 03:57:45.633884 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42633 (local address 127.0.0.1:34550)
0927 03:57:45.629757 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.629765 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.629802 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.630167 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.630167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.630169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.630199 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.630200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631628 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.631629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.632576 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.632579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633560 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.633643 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633656 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.633728 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.633786 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260927 03:57:45.633900 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34987 (local address 127.0.0.1:49682)
0927 03:57:45.629836 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.629845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.629898 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.630060 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.630060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.630062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.630136 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.630138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631221 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.631222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.632592 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.632595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633675 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.633737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633742 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.633789 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.633828 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":263,"thread_start_us":75,"threads_started":1}
I20260927 03:57:45.634143 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.1.50.66:41211)
0927 03:57:45.629614 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.629619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.629827 (+ 208us) server_negotiation.cc:407] Connection header received
0927 03:57:45.629872 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.629874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.629890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.629954 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.630324 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631406 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.632837 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633521 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633594 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633686 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.633946 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.634069 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 03:57:45.634296 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:41211 (local address 127.0.0.1:39618)
0927 03:57:45.629632 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.629672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.629746 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.630049 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.630052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.630055 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.630128 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.630130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631536 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.631538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.632766 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.632769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633726 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.633826 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633830 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.633872 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.633906 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":76,"threads_started":1}
I20260927 03:57:45.634567 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.629505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.1.50.65:34987)
0927 03:57:45.629789 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.629793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.629912 (+ 119us) server_negotiation.cc:407] Connection header received
0927 03:57:45.629956 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.629958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.629973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.630058 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.630212 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.631146 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.632682 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.633454 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.633531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.633625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.633937 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.634481 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":75,"threads_started":1}
I20260927 03:57:45.634785 21278 tablet_service.cc:1467] Processing CreateTablet for tablet 57c9190319a8481ba847c74df29de325 (DEFAULT_TABLE table=test-table [id=0e9ee32b8eba455da10cdfabc861ba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:45.634975 21147 tablet_service.cc:1467] Processing CreateTablet for tablet 57c9190319a8481ba847c74df29de325 (DEFAULT_TABLE table=test-table [id=0e9ee32b8eba455da10cdfabc861ba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:45.635165 21278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c9190319a8481ba847c74df29de325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:45.635272 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c9190319a8481ba847c74df29de325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:45.635241 21016 tablet_service.cc:1467] Processing CreateTablet for tablet 57c9190319a8481ba847c74df29de325 (DEFAULT_TABLE table=test-table [id=0e9ee32b8eba455da10cdfabc861ba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:57:45.635540 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c9190319a8481ba847c74df29de325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:45.637961 21358 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Bootstrap starting.
I20260927 03:57:45.638618 21358 tablet_bootstrap.cc:654] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:45.638581 21359 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Bootstrap starting.
I20260927 03:57:45.638707 21360 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Bootstrap starting.
I20260927 03:57:45.638933 21358 log.cc:824] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:45.639204 21359 tablet_bootstrap.cc:654] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:45.639305 21360 tablet_bootstrap.cc:654] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:45.639425 21359 log.cc:824] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:45.639619 21360 log.cc:824] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:45.639883 21358 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: No bootstrap required, opened a new log
I20260927 03:57:45.639955 21358 ts_tablet_manager.cc:1397] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:57:45.640007 21359 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: No bootstrap required, opened a new log
I20260927 03:57:45.640067 21359 ts_tablet_manager.cc:1397] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:45.640257 21360 tablet_bootstrap.cc:492] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: No bootstrap required, opened a new log
I20260927 03:57:45.640311 21360 ts_tablet_manager.cc:1397] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:57:45.642258 21360 raft_consensus.cc:348] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642395 21360 raft_consensus.cc:374] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:45.642426 21360 raft_consensus.cc:729] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94166f1e172d464ebc1153c020ce7dd7, State: Initialized, Role: FOLLOWER
I20260927 03:57:45.642572 21360 consensus_queue.cc:260] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642503 21359 raft_consensus.cc:348] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642566 21358 raft_consensus.cc:348] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642668 21359 raft_consensus.cc:374] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:45.642699 21359 raft_consensus.cc:729] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dc4a1ad9e14731a791c987bd9443ec, State: Initialized, Role: FOLLOWER
I20260927 03:57:45.642700 21358 raft_consensus.cc:374] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:45.642740 21358 raft_consensus.cc:729] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c82e12f07b4b329e7a1aa19506d443, State: Initialized, Role: FOLLOWER
I20260927 03:57:45.642834 21360 ts_tablet_manager.cc:1428] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:57:45.642807 21359 consensus_queue.cc:260] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642875 21358 consensus_queue.cc:260] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.642947 21082 heartbeater.cc:503] Master 127.1.50.126:45739 was elected leader, sending a full tablet report...
I20260927 03:57:45.643056 21359 ts_tablet_manager.cc:1428] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:57:45.643105 21358 ts_tablet_manager.cc:1428] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:57:45.643172 21344 heartbeater.cc:503] Master 127.1.50.126:45739 was elected leader, sending a full tablet report...
I20260927 03:57:45.643342 21213 heartbeater.cc:503] Master 127.1.50.126:45739 was elected leader, sending a full tablet report...
W20260927 03:57:45.717227 21214 tablet.cc:2369] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:45.832190 21364 raft_consensus.cc:482] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:57:45.832305 21364 raft_consensus.cc:504] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.832679 21364 leader_election.cc:288] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4dc4a1ad9e14731a791c987bd9443ec (127.1.50.67:42633), 45c82e12f07b4b329e7a1aa19506d443 (127.1.50.66:41211)
I20260927 03:57:45.835692 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.832980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:42067 (local address 127.1.50.66:41211)
0927 03:57:45.833072 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.833074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.833211 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:57:45.833262 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.833263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.833276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.833335 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.833474 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.833963 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.834993 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.835366 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.835432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.835441 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.835581 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.835620 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:57:45.835695 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.833008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.50.65:35115 (local address 127.1.50.67:42633)
0927 03:57:45.833052 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.833053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.833065 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:57:45.833104 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.833105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.833114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.833177 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:57:45.833348 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.833887 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.835002 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.835358 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.835416 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.835428 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.835594 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.835632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:57:45.835726 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.832900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.67:42633 (local address 127.1.50.65:35115)
0927 03:57:45.832957 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.832971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.833046 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.833203 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.833204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.833206 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.833282 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.833284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.833962 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.833963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.834901 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.834904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.835440 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.835441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.835528 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.835531 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.835582 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.835635 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":20}
I20260927 03:57:45.835737 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.832909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.66:41211 (local address 127.1.50.65:42067)
0927 03:57:45.833154 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.833163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:57:45.833216 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:57:45.833342 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:57:45.833343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:57:45.833344 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:57:45.833410 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.833412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.834105 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.834106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.834904 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:57:45.834907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.835442 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.835443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:57:45.835514 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.835517 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:57:45.835558 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:57:45.835602 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":83,"thread_start_us":84,"threads_started":1}
I20260927 03:57:45.836057 21298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c9190319a8481ba847c74df29de325" candidate_uuid: "94166f1e172d464ebc1153c020ce7dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" is_pre_election: true
I20260927 03:57:45.836117 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c9190319a8481ba847c74df29de325" candidate_uuid: "94166f1e172d464ebc1153c020ce7dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c82e12f07b4b329e7a1aa19506d443" is_pre_election: true
I20260927 03:57:45.836216 21298 raft_consensus.cc:2454] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94166f1e172d464ebc1153c020ce7dd7 in term 0.
I20260927 03:57:45.836259 21167 raft_consensus.cc:2454] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94166f1e172d464ebc1153c020ce7dd7 in term 0.
I20260927 03:57:45.836421 20969 leader_election.cc:302] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94166f1e172d464ebc1153c020ce7dd7, f4dc4a1ad9e14731a791c987bd9443ec; no voters:
I20260927 03:57:45.836614 21364 raft_consensus.cc:2781] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:57:45.836689 21364 raft_consensus.cc:482] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:57:45.836714 21364 raft_consensus.cc:3037] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.837437 21364 raft_consensus.cc:504] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.837566 21364 leader_election.cc:288] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [CANDIDATE]: Term 1 election: Requested vote from peers f4dc4a1ad9e14731a791c987bd9443ec (127.1.50.67:42633), 45c82e12f07b4b329e7a1aa19506d443 (127.1.50.66:41211)
I20260927 03:57:45.837738 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c9190319a8481ba847c74df29de325" candidate_uuid: "94166f1e172d464ebc1153c020ce7dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c82e12f07b4b329e7a1aa19506d443"
I20260927 03:57:45.837745 21298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c9190319a8481ba847c74df29de325" candidate_uuid: "94166f1e172d464ebc1153c020ce7dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc4a1ad9e14731a791c987bd9443ec"
I20260927 03:57:45.837816 21298 raft_consensus.cc:3037] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.837817 21167 raft_consensus.cc:3037] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:45.838518 21298 raft_consensus.cc:2454] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94166f1e172d464ebc1153c020ce7dd7 in term 1.
I20260927 03:57:45.838591 21167 raft_consensus.cc:2454] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94166f1e172d464ebc1153c020ce7dd7 in term 1.
I20260927 03:57:45.838696 20969 leader_election.cc:302] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94166f1e172d464ebc1153c020ce7dd7, f4dc4a1ad9e14731a791c987bd9443ec; no voters:
I20260927 03:57:45.838780 21364 raft_consensus.cc:2781] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:45.838950 21364 raft_consensus.cc:686] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 94166f1e172d464ebc1153c020ce7dd7, State: Running, Role: LEADER
I20260927 03:57:45.839047 21364 consensus_queue.cc:237] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } }
I20260927 03:57:45.839866 20894 catalog_manager.cc:5495] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94166f1e172d464ebc1153c020ce7dd7 (127.1.50.65). New cstate: current_term: 1 leader_uuid: "94166f1e172d464ebc1153c020ce7dd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94166f1e172d464ebc1153c020ce7dd7" member_type: VOTER last_known_addr { host: "127.1.50.65" port: 34987 } health_report { overall_health: HEALTHY } } }
W20260927 03:57:45.844676 21083 tablet.cc:2369] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:57:45.855554 21345 tablet.cc:2369] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:57:45.860446 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.858228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.1.50.65:34987)
0927 03:57:45.858284 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:45.858286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:45.858294 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:57:45.858355 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:45.858356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:45.858376 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:45.858434 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:57:45.858536 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.858994 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.859778 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:45.859967 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:45.860022 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:45.860156 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:57:45.860253 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:57:45.860313 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:45.860329 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:57:45.860346 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:57:45.862268 21167 raft_consensus.cc:1261] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 1 FOLLOWER]: Refusing update from remote peer 94166f1e172d464ebc1153c020ce7dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:45.862282 21298 raft_consensus.cc:1261] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 1 FOLLOWER]: Refusing update from remote peer 94166f1e172d464ebc1153c020ce7dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:57:45.862673 21368 consensus_queue.cc:1035] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4dc4a1ad9e14731a791c987bd9443ec" member_type: VOTER last_known_addr { host: "127.1.50.67" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:45.862816 21364 consensus_queue.cc:1035] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c82e12f07b4b329e7a1aa19506d443" member_type: VOTER last_known_addr { host: "127.1.50.66" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:57:45.865199 21371 mvcc.cc:205] Tried to move back new op lower bound from 7333812084169437184 to 7333812084077568000. Current Snapshot: MvccSnapshot[applied={T|T < 7333812084169437184}]
I20260927 03:57:45.865734 21372 mvcc.cc:205] Tried to move back new op lower bound from 7333812084169437184 to 7333812084077568000. Current Snapshot: MvccSnapshot[applied={T|T < 7333812084169437184}]
I20260927 03:57:45.866004 21373 mvcc.cc:205] Tried to move back new op lower bound from 7333812084169437184 to 7333812084077568000. Current Snapshot: MvccSnapshot[applied={T|T < 7333812084169437184}]
I20260927 03:57:45.867952 20892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:45.868031 20892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:57:45.868707 20892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1adbe5503b4584bdbb2bf6e075a02b: Sending DeleteTablet for 3 replicas of tablet 57c9190319a8481ba847c74df29de325
I20260927 03:57:45.869041 21278 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c9190319a8481ba847c74df29de325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='slave'} at 127.0.0.1:34550
I20260927 03:57:45.869064 21016 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c9190319a8481ba847c74df29de325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='slave'} at 127.0.0.1:49682
I20260927 03:57:45.869252 21382 tablet_replica.cc:330] T 57c9190319a8481ba847c74df29de325 P 94166f1e172d464ebc1153c020ce7dd7: stopping tablet replica
I20260927 03:57:45.869285 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20954
I20260927 03:57:45.870047 21381 tablet_replica.cc:330] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: stopping tablet replica
I20260927 03:57:45.870139 21381 raft_consensus.cc:2229] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:45.870225 21381 raft_consensus.cc:2258] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:45.870966 21381 ts_tablet_manager.cc:1905] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:45.872001 21147 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c9190319a8481ba847c74df29de325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:57:45 UTC) from {username='slave'} at 127.0.0.1:39618
I20260927 03:57:45.872274 21383 tablet_replica.cc:330] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: stopping tablet replica
I20260927 03:57:45.872370 21383 raft_consensus.cc:2229] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:57:45.872470 21383 raft_consensus.cc:2258] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:57:45.872704 21381 ts_tablet_manager.cc:1918] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:45.872769 21381 log.cc:1196] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-2/wal/wals/57c9190319a8481ba847c74df29de325
I20260927 03:57:45.872848 21383 ts_tablet_manager.cc:1905] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:57:45.873018 21381 ts_tablet_manager.cc:1939] T 57c9190319a8481ba847c74df29de325 P f4dc4a1ad9e14731a791c987bd9443ec: Deleting consensus metadata
I20260927 03:57:45.873395 20878 catalog_manager.cc:4841] TS f4dc4a1ad9e14731a791c987bd9443ec (127.1.50.67:42633): tablet 57c9190319a8481ba847c74df29de325 (table test-table [id=0e9ee32b8eba455da10cdfabc861ba4f]) successfully deleted
I20260927 03:57:45.874292 21383 ts_tablet_manager.cc:1918] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:57:45.874351 21383 log.cc:1196] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Deleting WAL directory at /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481222547165-1225-0/minicluster-data/ts-1/wal/wals/57c9190319a8481ba847c74df29de325
I20260927 03:57:45.874577 21383 ts_tablet_manager.cc:1939] T 57c9190319a8481ba847c74df29de325 P 45c82e12f07b4b329e7a1aa19506d443: Deleting consensus metadata
I20260927 03:57:45.874930 20879 catalog_manager.cc:4841] TS 45c82e12f07b4b329e7a1aa19506d443 (127.1.50.66:41211): tablet 57c9190319a8481ba847c74df29de325 (table test-table [id=0e9ee32b8eba455da10cdfabc861ba4f]) successfully deleted
W20260927 03:57:45.875074 20881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.50.65:34987 (error 108)
I20260927 03:57:45.875255 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21085
I20260927 03:57:45.875783 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:45.875337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.65:34987 (local address 127.0.0.1:49688)
0927 03:57:45.875711 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 03:57:45.875749 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
W20260927 03:57:45.875910 20881 catalog_manager.cc:4584] TS 94166f1e172d464ebc1153c020ce7dd7 (127.1.50.65:34987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57c9190319a8481ba847c74df29de325: Network error: Client connection negotiation failed: client connection to 127.1.50.65:34987: connect: Connection refused (error 111)
I20260927 03:57:45.879848 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21216
I20260927 03:57:45.883704 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 20863
2026-09-27T03:57:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (831 ms)
[----------] 6 tests from EncryptionPolicyTest (8600 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:45 dist-test-slave-96fg krb5kdc[21387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:45 dist-test-slave-96fg krb5kdc[21387](info): set up 2 sockets
Sep 27 03:57:45 dist-test-slave-96fg krb5kdc[21387](info): commencing operation
krb5kdc: starting...
W20260927 03:57:46.837747 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:46 dist-test-slave-96fg krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:46.958137 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:47.039191 21400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:47.039435 21400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:47.039470 21400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:47.040988 21400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:47.041064 21400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:47.041088 21400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:47.041105 21400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:47.041121 21400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:47.043533 21400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:47.043860 21400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:47.044137 21400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:47.049470 21408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:47.049535 21405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:47.049547 21400 server_base.cc:1034] running on GCE node
W20260927 03:57:47.049470 21406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:47.049980 21400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:47.050343 21400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:47.051513 21400 hybrid_clock.cc:648] HybridClock initialized: now 1790481467051497 us; error 31 us; skew 500 ppm
Sep 27 03:57:47 dist-test-slave-96fg krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:47.054013 21400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:47.054493 21400 webserver.cc:460] Webserver started at http://127.0.0.1:41879/ using document root <none> and password file <none>
I20260927 03:57:47.054725 21400 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:47.054770 21400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:47.054886 21400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:47.055912 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "da29fec06b4145dfa24f59a49ea0f856"
format_stamp: "Formatted at 2026-09-27 03:57:47 on dist-test-slave-96fg"
server_key: "9b74900bc7b379bf8cdf2226484d6a95"
server_key_iv: "44f503c3bfa26866f824641725ac67bd"
server_key_version: "encryptionkey@0"
I20260927 03:57:47.056226 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "da29fec06b4145dfa24f59a49ea0f856"
format_stamp: "Formatted at 2026-09-27 03:57:47 on dist-test-slave-96fg"
server_key: "9b74900bc7b379bf8cdf2226484d6a95"
server_key_iv: "44f503c3bfa26866f824641725ac67bd"
server_key_version: "encryptionkey@0"
I20260927 03:57:47.057489 21400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:47.058272 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:47.058444 21400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:47.058533 21400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "da29fec06b4145dfa24f59a49ea0f856"
format_stamp: "Formatted at 2026-09-27 03:57:47 on dist-test-slave-96fg"
server_key: "9b74900bc7b379bf8cdf2226484d6a95"
server_key_iv: "44f503c3bfa26866f824641725ac67bd"
server_key_version: "encryptionkey@0"
I20260927 03:57:47.058595 21400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:47.069908 21400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:47.071796 21400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:47.071935 21400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:47.075736 21400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46709
I20260927 03:57:47.075769 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46709 every 8 connection(s)
I20260927 03:57:47.076098 21400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:47.076999 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:47.079241 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: Bootstrap starting.
I20260927 03:57:47.079737 21468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:47.079950 21468 log.cc:824] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:47.080633 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: No bootstrap required, opened a new log
I20260927 03:57:47.082348 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } }
I20260927 03:57:47.082489 21468 raft_consensus.cc:374] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:47.082509 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da29fec06b4145dfa24f59a49ea0f856, State: Initialized, Role: FOLLOWER
I20260927 03:57:47.082616 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } }
I20260927 03:57:47.082686 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:47.082724 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:47.082770 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:47.082870 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21400
I20260927 03:57:47.082983 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:47.083170 1225 external_mini_cluster.cc:1428] Setting key b15eba21ed995395a6f5080c626740bf
I20260927 03:57:47.083422 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } }
I20260927 03:57:47.083523 21468 leader_election.cc:302] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da29fec06b4145dfa24f59a49ea0f856; no voters:
I20260927 03:57:47.083702 21468 leader_election.cc:288] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:47.083796 21472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:47.083904 21472 raft_consensus.cc:686] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [term 1 LEADER]: Becoming Leader. State: Replica: da29fec06b4145dfa24f59a49ea0f856, State: Running, Role: LEADER
I20260927 03:57:47.084007 21472 consensus_queue.cc:237] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } }
I20260927 03:57:47.084009 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:47.084820 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da29fec06b4145dfa24f59a49ea0f856. Latest consensus state: current_term: 1 leader_uuid: "da29fec06b4145dfa24f59a49ea0f856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } } }
I20260927 03:57:47.085189 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:47.085213 21475 sys_catalog.cc:455] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da29fec06b4145dfa24f59a49ea0f856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da29fec06b4145dfa24f59a49ea0f856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46709 } } }
I20260927 03:57:47.085321 21475 sys_catalog.cc:458] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856 [sys.catalog]: This master's current role is: LEADER
W20260927 03:57:47.085904 21489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:47.085950 21489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:47.086033 21488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:47.086216 21488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:47.087487 21488 catalog_manager.cc:1347] Generated new cluster ID: 027fb09c9551421eab19826006215732
I20260927 03:57:47.087532 21488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:57:47 dist-test-slave-96fg krb5kdc[21387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:57:47.091292 21488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:47.091795 21488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:47.097106 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:47.083613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57541 (local address 127.0.0.1:46709)
0927 03:57:47.083771 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:47.083782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:47.083820 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:57:47.083943 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:47.083950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:47.084014 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:47.084184 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:47.084939 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:47.086327 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:47.087343 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:47.087948 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:47.088048 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:47.090956 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:47.090960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:47.090961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:47.090974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:47.095500 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:47.095953 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:47.095956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:47.095956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:47.096054 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:47.096264 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:47.096265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:47.096265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:47.096522 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:47.096691 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:47.096788 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 03:57:47.096882 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260927 03:57:47.097965 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:47.098819 21488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da29fec06b4145dfa24f59a49ea0f856: Generated new TSK 0
I20260927 03:57:47.098973 21488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:47.105866 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:47.099421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46645 (local address 127.0.0.1:46709)
0927 03:57:47.099469 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:47.099471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:47.099536 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:57:47.099623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:47.099631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:47.099641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:47.099703 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:47.100138 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:47.100929 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:47.101718 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:47.102081 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:47.102146 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:47.103262 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:47.103264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:47.103264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:47.103272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:47.104739 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:47.105085 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:47.105086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:47.105086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:47.105143 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:47.105320 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:47.105320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:47.105321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:47.105443 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:47.105542 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:47.105709 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:57:47.105771 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:57:47.107839 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21400
W20260927 03:57:47.111510 21385 connection.cc:535] client connection to 127.0.0.1:46709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46709): Connection reset by peer (error 104)
2026-09-27T03:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:47 dist-test-slave-96fg krb5kdc[21499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:47 dist-test-slave-96fg krb5kdc[21499](info): set up 2 sockets
Sep 27 03:57:47 dist-test-slave-96fg krb5kdc[21499](info): commencing operation
krb5kdc: starting...
W20260927 03:57:48.069764 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:48 dist-test-slave-96fg krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:48.189249 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:48.268363 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:48.268576 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:48.268599 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:48.269848 21512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:48.269891 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:48.269903 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:48.269914 21512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:48.269925 21512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:48.271427 21512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:48.271622 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:48.271786 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:48.276438 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:48.276441 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:48.276432 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:48.276688 21512 server_base.cc:1034] running on GCE node
I20260927 03:57:48.276948 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:48.277200 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:48.278340 21512 hybrid_clock.cc:648] HybridClock initialized: now 1790481468278323 us; error 29 us; skew 500 ppm
Sep 27 03:57:48 dist-test-slave-96fg krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:48.286000 21512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:48.286590 21512 webserver.cc:460] Webserver started at http://127.0.0.1:43337/ using document root <none> and password file <none>
I20260927 03:57:48.286818 21512 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:48.286867 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:48.286986 21512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:48.288013 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "5052ea8f46ec45509de93ad2774ab295"
format_stamp: "Formatted at 2026-09-27 03:57:48 on dist-test-slave-96fg"
server_key: "fcfb525e56a604893e7d9f109587e93c"
server_key_iv: "848d62ffe04550bd8470ff9bedfdcd5b"
server_key_version: "encryptionkey@0"
I20260927 03:57:48.288347 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "5052ea8f46ec45509de93ad2774ab295"
format_stamp: "Formatted at 2026-09-27 03:57:48 on dist-test-slave-96fg"
server_key: "fcfb525e56a604893e7d9f109587e93c"
server_key_iv: "848d62ffe04550bd8470ff9bedfdcd5b"
server_key_version: "encryptionkey@0"
I20260927 03:57:48.289678 21512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:48.290601 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:48.290776 21512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:57:48.290867 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "5052ea8f46ec45509de93ad2774ab295"
format_stamp: "Formatted at 2026-09-27 03:57:48 on dist-test-slave-96fg"
server_key: "fcfb525e56a604893e7d9f109587e93c"
server_key_iv: "848d62ffe04550bd8470ff9bedfdcd5b"
server_key_version: "encryptionkey@0"
I20260927 03:57:48.290936 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:48.301829 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:48.304101 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:48.304250 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:48.308243 21512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37057
I20260927 03:57:48.308279 21579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37057 every 8 connection(s)
I20260927 03:57:48.308604 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:48.309549 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:48.311744 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295: Bootstrap starting.
I20260927 03:57:48.312247 21580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:48.312460 21580 log.cc:824] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:48.313058 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295: No bootstrap required, opened a new log
I20260927 03:57:48.313961 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21512
I20260927 03:57:48.314097 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:48.314283 1225 external_mini_cluster.cc:1428] Setting key d6d178747c8c2ea31457b53abfadc316
I20260927 03:57:48.314988 21580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } }
I20260927 03:57:48.315136 21580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:48.315158 21580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5052ea8f46ec45509de93ad2774ab295, State: Initialized, Role: FOLLOWER
I20260927 03:57:48.315268 21580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } }
I20260927 03:57:48.315335 21580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:48.315398 21580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:48.315452 21580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:48.315980 21580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } }
I20260927 03:57:48.316069 21580 leader_election.cc:302] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5052ea8f46ec45509de93ad2774ab295; no voters:
I20260927 03:57:48.316222 21580 leader_election.cc:288] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:48.316262 21585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:48.316370 21585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [term 1 LEADER]: Becoming Leader. State: Replica: 5052ea8f46ec45509de93ad2774ab295, State: Running, Role: LEADER
I20260927 03:57:48.316468 21580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:48.316509 21585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } }
I20260927 03:57:48.316874 21585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5052ea8f46ec45509de93ad2774ab295. Latest consensus state: current_term: 1 leader_uuid: "5052ea8f46ec45509de93ad2774ab295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } } }
I20260927 03:57:48.316931 21587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5052ea8f46ec45509de93ad2774ab295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5052ea8f46ec45509de93ad2774ab295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37057 } } }
I20260927 03:57:48.317005 21585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:48.317018 21587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:48.317221 21594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:48.317745 21594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:48.319204 21594 catalog_manager.cc:1347] Generated new cluster ID: 4116ea0033e445138b9b8c30c5607a69
I20260927 03:57:48.319258 21594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:48.319988 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:48.314794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40349 (local address 127.0.0.1:37057)
0927 03:57:48.314970 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:48.314977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:48.315002 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:57:48.315146 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:48.315153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:48.315210 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:48.315349 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:48.315679 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:48.316791 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:48.317847 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:48.318450 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:48.319075 (+ 625us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:48.319364 (+ 289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:48.319367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:48.319368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:48.319521 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:48.319550 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:48.319560 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:48.319625 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:48.319719 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:57:48.319832 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260927 03:57:48.321115 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:48.325307 21594 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:48.325546 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:48.322483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37539 (local address 127.0.0.1:37057)
0927 03:57:48.322527 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:48.322529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:48.322593 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:57:48.322666 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:48.322667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:48.322679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:48.322738 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:48.322864 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:48.323355 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:48.324070 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:48.324351 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:48.324412 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:48.325196 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:48.325197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:48.325198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:48.325203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:48.325218 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:48.325223 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:48.325276 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:48.325442 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:57:48.325457 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":25}
I20260927 03:57:48.325979 21594 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:48.330508 21594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5052ea8f46ec45509de93ad2774ab295: Generated new TSK 0
I20260927 03:57:48.330667 21594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:48.334569 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21512
W20260927 03:57:48.338317 21497 connection.cc:535] client connection to 127.0.0.1:37057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37057): Connection reset by peer (error 104)
2026-09-27T03:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:48 dist-test-slave-96fg krb5kdc[21611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:48 dist-test-slave-96fg krb5kdc[21611](info): set up 2 sockets
Sep 27 03:57:48 dist-test-slave-96fg krb5kdc[21611](info): commencing operation
krb5kdc: starting...
W20260927 03:57:49.285733 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:49 dist-test-slave-96fg krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:49.405324 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:49.488559 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:49.488821 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:49.488852 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:49.490052 21624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:49.490113 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:49.490135 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:49.490152 21624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:49.490170 21624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:49.491695 21624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:49.491921 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:49.492138 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:49.496765 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:49.496903 21624 server_base.cc:1034] running on GCE node
W20260927 03:57:49.496814 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:49.496820 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:49.497213 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:49.497484 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:49.498633 21624 hybrid_clock.cc:648] HybridClock initialized: now 1790481469498613 us; error 31 us; skew 500 ppm
Sep 27 03:57:49 dist-test-slave-96fg krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:49.501098 21624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:49.501509 21624 webserver.cc:460] Webserver started at http://127.0.0.1:46619/ using document root <none> and password file <none>
I20260927 03:57:49.501744 21624 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:49.501792 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:49.501912 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:49.503129 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "a0fd6387fde74171809fc0d9f77e696e"
format_stamp: "Formatted at 2026-09-27 03:57:49 on dist-test-slave-96fg"
server_key: "94b8c09bf46052471b748b15ea0e7186"
server_key_iv: "6cc24aa0187ed5a434ef129645f71539"
server_key_version: "encryptionkey@0"
I20260927 03:57:49.503475 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "a0fd6387fde74171809fc0d9f77e696e"
format_stamp: "Formatted at 2026-09-27 03:57:49 on dist-test-slave-96fg"
server_key: "94b8c09bf46052471b748b15ea0e7186"
server_key_iv: "6cc24aa0187ed5a434ef129645f71539"
server_key_version: "encryptionkey@0"
I20260927 03:57:49.504798 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:49.505475 21639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:49.505659 21624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:57:49.505743 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "a0fd6387fde74171809fc0d9f77e696e"
format_stamp: "Formatted at 2026-09-27 03:57:49 on dist-test-slave-96fg"
server_key: "94b8c09bf46052471b748b15ea0e7186"
server_key_iv: "6cc24aa0187ed5a434ef129645f71539"
server_key_version: "encryptionkey@0"
I20260927 03:57:49.505807 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:49.509011 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:49.511046 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:49.511183 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:49.514716 21624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36399
I20260927 03:57:49.514757 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36399 every 8 connection(s)
I20260927 03:57:49.515038 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:49.515893 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:49.518092 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e: Bootstrap starting.
I20260927 03:57:49.518630 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:49.518870 21692 log.cc:824] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:49.519511 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e: No bootstrap required, opened a new log
I20260927 03:57:49.520069 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21624
I20260927 03:57:49.520187 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:49.520368 1225 external_mini_cluster.cc:1428] Setting key be92eab1de4a786d315ea13fc0245bac
I20260927 03:57:49.521430 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 03:57:49.521564 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:49.521596 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0fd6387fde74171809fc0d9f77e696e, State: Initialized, Role: FOLLOWER
I20260927 03:57:49.521750 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 03:57:49.521853 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:49.521883 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:49.521929 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:49.522810 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:49.520869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41737 (local address 127.0.0.1:36399)
0927 03:57:49.521034 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:49.521042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:49.521066 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:57:49.521402 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:49.521408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:49.521432 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:49.521559 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:49.522301 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:49.522305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:49.522305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:49.522309 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:49.522345 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:49.522356 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:49.522411 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:49.522584 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 03:57:49.522634 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260927 03:57:49.522891 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 03:57:49.523003 21692 leader_election.cc:302] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0fd6387fde74171809fc0d9f77e696e; no voters:
I20260927 03:57:49.523172 21692 leader_election.cc:288] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:49.523214 21697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:49.523324 21697 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [term 1 LEADER]: Becoming Leader. State: Replica: a0fd6387fde74171809fc0d9f77e696e, State: Running, Role: LEADER
I20260927 03:57:49.523449 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:49.523447 21697 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 03:57:49.523631 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:49.523872 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0fd6387fde74171809fc0d9f77e696e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } } }
I20260927 03:57:49.524098 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:49.524024 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0fd6387fde74171809fc0d9f77e696e. Latest consensus state: current_term: 1 leader_uuid: "a0fd6387fde74171809fc0d9f77e696e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fd6387fde74171809fc0d9f77e696e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } } }
I20260927 03:57:49.524546 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:49.524715 21706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:49.525210 21706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:49.526543 21706 catalog_manager.cc:1347] Generated new cluster ID: 93a10cb931014aa5a12cc2641e5386a4
I20260927 03:57:49.526593 21706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:49.531252 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:49.525140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59093 (local address 127.0.0.1:36399)
0927 03:57:49.525171 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:49.525173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:49.530246 (+ 5073us) server_negotiation.cc:407] Connection header received
0927 03:57:49.530322 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:49.530323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:49.530330 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:49.530387 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:49.531037 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:49.531038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:49.531039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:49.531043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:49.531058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:49.531062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:49.531116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:49.531168 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:57:49.531183 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:57:49.540640 21706 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:49.540844 21706 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:49.545809 21706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0fd6387fde74171809fc0d9f77e696e: Generated new TSK 0
I20260927 03:57:49.546067 21706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:49.549605 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21624
2026-09-27T03:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:49 dist-test-slave-96fg krb5kdc[21723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:49 dist-test-slave-96fg krb5kdc[21723](info): set up 2 sockets
Sep 27 03:57:49 dist-test-slave-96fg krb5kdc[21723](info): commencing operation
krb5kdc: starting...
W20260927 03:57:50.513751 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:50.633749 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:50.713078 21736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:50.713286 21736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:50.713316 21736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:50.714529 21736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:50.714579 21736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:50.714605 21736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:50.714622 21736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:50.714637 21736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:50.716135 21736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:50.716379 21736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:50.716586 21736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:50.721587 21744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:50.721576 21741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:50.721668 21742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:50.721753 21736 server_base.cc:1034] running on GCE node
I20260927 03:57:50.722093 21736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:50.722400 21736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:50.723547 21736 hybrid_clock.cc:648] HybridClock initialized: now 1790481470723535 us; error 27 us; skew 500 ppm
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:50.725893 21736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:50.726377 21736 webserver.cc:460] Webserver started at http://127.0.0.1:33991/ using document root <none> and password file <none>
I20260927 03:57:50.726586 21736 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:50.726630 21736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:50.726735 21736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:50.727809 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "9307345815434b748dbaaa6321e738ca"
format_stamp: "Formatted at 2026-09-27 03:57:50 on dist-test-slave-96fg"
server_key: "9d9a50335af559d2f2cdc160d26eb0fb"
server_key_iv: "8459afcb7af889f973425efbb0cd70fa"
server_key_version: "encryptionkey@0"
I20260927 03:57:50.728121 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "9307345815434b748dbaaa6321e738ca"
format_stamp: "Formatted at 2026-09-27 03:57:50 on dist-test-slave-96fg"
server_key: "9d9a50335af559d2f2cdc160d26eb0fb"
server_key_iv: "8459afcb7af889f973425efbb0cd70fa"
server_key_version: "encryptionkey@0"
I20260927 03:57:50.729375 21736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:57:50.730159 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:50.730302 21736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:50.730376 21736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9307345815434b748dbaaa6321e738ca"
format_stamp: "Formatted at 2026-09-27 03:57:50 on dist-test-slave-96fg"
server_key: "9d9a50335af559d2f2cdc160d26eb0fb"
server_key_iv: "8459afcb7af889f973425efbb0cd70fa"
server_key_version: "encryptionkey@0"
I20260927 03:57:50.730443 21736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:50.751761 21736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:50.753672 21736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:50.753809 21736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:50.757642 21736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36455
I20260927 03:57:50.757695 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36455 every 8 connection(s)
I20260927 03:57:50.758005 21736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:50.758831 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21736
I20260927 03:57:50.758957 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:50.759038 21804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:50.759143 1225 external_mini_cluster.cc:1428] Setting key b7b07a1970df73f8d8e7eb4af8449ad1
I20260927 03:57:50.762285 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca: Bootstrap starting.
I20260927 03:57:50.763001 21804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:50.763319 21804 log.cc:824] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:50.764055 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca: No bootstrap required, opened a new log
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:57:50.766814 21804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260927 03:57:50.767026 21804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:50.767062 21804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9307345815434b748dbaaa6321e738ca, State: Initialized, Role: FOLLOWER
I20260927 03:57:50.767274 21804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260927 03:57:50.767392 21804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:50.767450 21804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:50.767521 21804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:50.768307 21804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260927 03:57:50.768456 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9307345815434b748dbaaa6321e738ca; no voters:
I20260927 03:57:50.768646 21804 leader_election.cc:288] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:50.768841 21810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:50.768926 21804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:50.769037 21810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [term 1 LEADER]: Becoming Leader. State: Replica: 9307345815434b748dbaaa6321e738ca, State: Running, Role: LEADER
I20260927 03:57:50.769137 21810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260927 03:57:50.769477 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9307345815434b748dbaaa6321e738ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } } }
I20260927 03:57:50.769572 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:50.769702 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9307345815434b748dbaaa6321e738ca. Latest consensus state: current_term: 1 leader_uuid: "9307345815434b748dbaaa6321e738ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9307345815434b748dbaaa6321e738ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } } }
I20260927 03:57:50.769769 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:50.770166 21814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:50.770727 21814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:50.771234 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:50.759657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55831 (local address 127.0.0.1:36455)
0927 03:57:50.759819 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:50.759828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:50.759852 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:57:50.760456 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:50.760462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:50.760510 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:50.760628 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:50.761435 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:50.762432 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:50.763456 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:50.763907 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:50.764017 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:50.766905 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:50.766910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:50.766911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:50.766926 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:50.769217 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:50.769701 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:50.769704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:50.769704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:50.769813 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:50.770119 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:50.770120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:50.770120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:50.770442 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:50.770602 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:50.770759 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:57:50.771032 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260927 03:57:50.772260 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:50.775673 21814 catalog_manager.cc:1347] Generated new cluster ID: b8329e9357a9472f93afc7555968a9dc
I20260927 03:57:50.775732 21814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:50.785493 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:50.778166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47417 (local address 127.0.0.1:36455)
0927 03:57:50.778223 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:50.778225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:50.778236 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:57:50.778261 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:50.778262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:50.778273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:50.778343 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:50.778998 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:50.779749 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:50.780777 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:50.781087 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:50.781153 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:50.782608 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:50.782609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:50.782610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:50.782616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:50.784217 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:50.784615 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:50.784616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:50.784616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:50.784676 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:50.784853 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:50.784853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:50.784854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:50.784980 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:50.785075 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:50.785288 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:57:50.785380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:57:50.792682 21814 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:50.793313 21814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:50.798090 21814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9307345815434b748dbaaa6321e738ca: Generated new TSK 0
I20260927 03:57:50.798280 21814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:50.803303 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21736
2026-09-27T03:57:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21835](info): set up 2 sockets
Sep 27 03:57:50 dist-test-slave-96fg krb5kdc[21835](info): commencing operation
krb5kdc: starting...
W20260927 03:57:51.758139 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:51 dist-test-slave-96fg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:51.890681 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:51.970249 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:51.970479 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:51.970511 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:51.971751 21848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:51.971801 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:51.971819 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:51.971836 21848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:51.971851 21848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:51.973417 21848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:51.973666 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:51.973884 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:51.978895 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:51.978931 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:51.978952 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:51.979015 21848 server_base.cc:1034] running on GCE node
I20260927 03:57:51.979352 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:51.979627 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:51.980779 21848 hybrid_clock.cc:648] HybridClock initialized: now 1790481471980765 us; error 27 us; skew 500 ppm
Sep 27 03:57:51 dist-test-slave-96fg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:51.983296 21848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:51.983753 21848 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
I20260927 03:57:51.983978 21848 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:51.984020 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:51.984135 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:51.985208 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "1d0d7edb48ae4139a7ef6848df89aaee"
format_stamp: "Formatted at 2026-09-27 03:57:51 on dist-test-slave-96fg"
server_key: "97371b5013793248d8401bd9254308df"
server_key_iv: "6cbb9a8b958a5db8862223f62158bf1c"
server_key_version: "encryptionkey@0"
I20260927 03:57:51.985536 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "1d0d7edb48ae4139a7ef6848df89aaee"
format_stamp: "Formatted at 2026-09-27 03:57:51 on dist-test-slave-96fg"
server_key: "97371b5013793248d8401bd9254308df"
server_key_iv: "6cbb9a8b958a5db8862223f62158bf1c"
server_key_version: "encryptionkey@0"
I20260927 03:57:51.986856 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:51.987637 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:51.987825 21848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:51.987912 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "1d0d7edb48ae4139a7ef6848df89aaee"
format_stamp: "Formatted at 2026-09-27 03:57:51 on dist-test-slave-96fg"
server_key: "97371b5013793248d8401bd9254308df"
server_key_iv: "6cbb9a8b958a5db8862223f62158bf1c"
server_key_version: "encryptionkey@0"
I20260927 03:57:51.987970 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:52.007856 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:52.010413 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:52.010571 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:52.014482 21848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43041
I20260927 03:57:52.014518 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43041 every 8 connection(s)
I20260927 03:57:52.014870 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:52.015532 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21848
I20260927 03:57:52.015640 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:52.015825 1225 external_mini_cluster.cc:1428] Setting key bd1d317a39531862f26a31f30f6922f5
I20260927 03:57:52.015929 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:52.019356 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee: Bootstrap starting.
I20260927 03:57:52.020020 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:52.020289 21916 log.cc:824] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:52.020992 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee: No bootstrap required, opened a new log
I20260927 03:57:52.021804 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:52.016392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59253 (local address 127.0.0.1:43041)
0927 03:57:52.016569 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:52.016584 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:52.016608 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:57:52.017283 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:52.017290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:52.017338 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:52.017468 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:52.017780 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:52.018728 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:52.019787 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:52.020286 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:52.020388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:52.021036 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:52.021041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:52.021041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:52.021306 (+ 265us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:52.021344 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:52.021360 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:52.021434 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:52.021509 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:57:52.021579 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260927 03:57:52.022924 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260927 03:57:52.023065 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:52.023088 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0d7edb48ae4139a7ef6848df89aaee, State: Initialized, Role: FOLLOWER
I20260927 03:57:52.023169 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:52.023211 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260927 03:57:52.023298 21916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:52.023332 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:52.023378 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:52.023944 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260927 03:57:52.024030 21916 leader_election.cc:302] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0d7edb48ae4139a7ef6848df89aaee; no voters:
I20260927 03:57:52.024282 21916 leader_election.cc:288] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:52.024331 21925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:52.024477 21925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [term 1 LEADER]: Becoming Leader. State: Replica: 1d0d7edb48ae4139a7ef6848df89aaee, State: Running, Role: LEADER
I20260927 03:57:52.024557 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:52.024600 21925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260927 03:57:52.024978 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } } }
I20260927 03:57:52.024999 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0d7edb48ae4139a7ef6848df89aaee. Latest consensus state: current_term: 1 leader_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0d7edb48ae4139a7ef6848df89aaee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } } }
I20260927 03:57:52.025056 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:52.025070 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:52.025328 21933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:52.025799 21933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:52.027875 21933 catalog_manager.cc:1347] Generated new cluster ID: cfcfa3e94d694a0d9d8ecd93e1a2f458
I20260927 03:57:52.027978 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:52.024500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54795 (local address 127.0.0.1:43041)
0927 03:57:52.024535 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:52.024537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:52.024810 (+ 273us) server_negotiation.cc:407] Connection header received
0927 03:57:52.024857 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:52.024858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:52.024869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:52.024929 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:52.025081 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:52.025786 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:52.026672 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:52.026944 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:52.027016 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:52.027700 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:52.027701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:52.027701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:52.027707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:52.027731 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:52.027735 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:52.027787 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:52.027883 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:57:52.027909 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:57:52.027982 21933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:52.037576 21933 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:52.038424 21933 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:52.056706 21933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d0d7edb48ae4139a7ef6848df89aaee: Generated new TSK 0
I20260927 03:57:52.056908 21933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:52.063215 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21848
2026-09-27T03:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:52 dist-test-slave-96fg krb5kdc[21947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:52 dist-test-slave-96fg krb5kdc[21947](info): set up 2 sockets
Sep 27 03:57:52 dist-test-slave-96fg krb5kdc[21947](info): commencing operation
krb5kdc: starting...
W20260927 03:57:53.021827 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:53 dist-test-slave-96fg krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:53.142138 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:53.221688 21960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:53.221916 21960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:53.221937 21960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:53.223081 21960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:53.223125 21960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:53.223138 21960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:53.223150 21960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:53.223160 21960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:53.224618 21960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:53.224812 21960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:53.224997 21960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:53.229785 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:53.229825 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:53.229782 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:53.230034 21960 server_base.cc:1034] running on GCE node
I20260927 03:57:53.230320 21960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:53.230583 21960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:53.231732 21960 hybrid_clock.cc:648] HybridClock initialized: now 1790481473231714 us; error 34 us; skew 500 ppm
Sep 27 03:57:53 dist-test-slave-96fg krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:53.234163 21960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:53.234618 21960 webserver.cc:460] Webserver started at http://127.0.0.1:42071/ using document root <none> and password file <none>
I20260927 03:57:53.234886 21960 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:53.234941 21960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:53.235065 21960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:53.236133 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "537096f95a4e464dbfcef28690e3e938"
format_stamp: "Formatted at 2026-09-27 03:57:53 on dist-test-slave-96fg"
server_key: "5550112d22b941ef78712ce1914faa3f"
server_key_iv: "4b3cdb75a5ade62da9248114e1a48dd2"
server_key_version: "encryptionkey@0"
I20260927 03:57:53.236482 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "537096f95a4e464dbfcef28690e3e938"
format_stamp: "Formatted at 2026-09-27 03:57:53 on dist-test-slave-96fg"
server_key: "5550112d22b941ef78712ce1914faa3f"
server_key_iv: "4b3cdb75a5ade62da9248114e1a48dd2"
server_key_version: "encryptionkey@0"
I20260927 03:57:53.237743 21960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:57:53.238525 21975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:53.238698 21960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:53.238782 21960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "537096f95a4e464dbfcef28690e3e938"
format_stamp: "Formatted at 2026-09-27 03:57:53 on dist-test-slave-96fg"
server_key: "5550112d22b941ef78712ce1914faa3f"
server_key_iv: "4b3cdb75a5ade62da9248114e1a48dd2"
server_key_version: "encryptionkey@0"
I20260927 03:57:53.238849 21960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:53.242143 21960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:53.243980 21960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:53.244123 21960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:53.247896 21960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34673
I20260927 03:57:53.247948 22027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34673 every 8 connection(s)
I20260927 03:57:53.248247 21960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:53.249233 22028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:53.251466 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: Bootstrap starting.
I20260927 03:57:53.252004 22028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:53.252262 22028 log.cc:824] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:53.252946 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: No bootstrap required, opened a new log
I20260927 03:57:53.254631 22028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } }
I20260927 03:57:53.254769 22028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:53.254791 22028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537096f95a4e464dbfcef28690e3e938, State: Initialized, Role: FOLLOWER
I20260927 03:57:53.254916 22028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } }
I20260927 03:57:53.254995 22028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:53.255021 22028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:53.255067 22028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:53.255577 22028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } }
I20260927 03:57:53.255666 22028 leader_election.cc:302] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537096f95a4e464dbfcef28690e3e938; no voters:
I20260927 03:57:53.255837 22028 leader_election.cc:288] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:53.255926 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:53.256070 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [term 1 LEADER]: Becoming Leader. State: Replica: 537096f95a4e464dbfcef28690e3e938, State: Running, Role: LEADER
I20260927 03:57:53.256089 22028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:53.256199 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } }
I20260927 03:57:53.256537 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537096f95a4e464dbfcef28690e3e938. Latest consensus state: current_term: 1 leader_uuid: "537096f95a4e464dbfcef28690e3e938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } } }
I20260927 03:57:53.256633 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:53.256735 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 21960
I20260927 03:57:53.256848 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:53.256740 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "537096f95a4e464dbfcef28690e3e938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537096f95a4e464dbfcef28690e3e938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34673 } } }
I20260927 03:57:53.256892 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:53.257021 1225 external_mini_cluster.cc:1428] Setting key 7f7a3b0708936bc5525b06cbbb658015
W20260927 03:57:53.258212 22049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:53.258311 22049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:53.258414 22035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:53.258596 22035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:53.259647 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:53.257487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42263 (local address 127.0.0.1:34673)
0927 03:57:53.257626 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:53.257635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:53.257824 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:57:53.257988 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:53.257994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:53.258050 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:53.258193 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:53.259257 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:53.259260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:53.259260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:53.259264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:53.259302 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:53.259312 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:53.259365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:53.259429 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:57:53.259466 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":67,"threads_started":1}
I20260927 03:57:53.260016 22035 catalog_manager.cc:1347] Generated new cluster ID: 7f4715c9c28a4f6692331b67ff575799
I20260927 03:57:53.260062 22035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:53.260381 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:53.267541 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:53.266424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57893 (local address 127.0.0.1:34673)
0927 03:57:53.266465 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:53.266467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:53.266616 (+ 149us) server_negotiation.cc:407] Connection header received
0927 03:57:53.266677 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:53.266678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:53.266685 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:53.266733 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:53.267287 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:53.267288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:53.267288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:53.267293 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:53.267309 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:53.267314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:53.267364 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:53.267460 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:57:53.267474 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:57:53.267765 22035 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:53.267940 22035 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:53.272325 22035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 537096f95a4e464dbfcef28690e3e938: Generated new TSK 0
I20260927 03:57:53.272531 22035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:53.275182 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 21960
2026-09-27T03:57:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:53 dist-test-slave-96fg krb5kdc[22059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:53 dist-test-slave-96fg krb5kdc[22059](info): set up 2 sockets
Sep 27 03:57:53 dist-test-slave-96fg krb5kdc[22059](info): commencing operation
krb5kdc: starting...
W20260927 03:57:54.233909 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:54.357033 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:54.442504 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:54.442741 22072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:54.442765 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:54.444015 22072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:54.444067 22072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:54.444080 22072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:54.444092 22072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:54.444104 22072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:54.445673 22072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:54.445932 22072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:54.446194 22072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:54.451480 22077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:54.451489 22080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:54.451747 22072 server_base.cc:1034] running on GCE node
W20260927 03:57:54.451512 22078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:54.452178 22072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:54.452522 22072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:54.453699 22072 hybrid_clock.cc:648] HybridClock initialized: now 1790481474453679 us; error 40 us; skew 500 ppm
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:54.456912 22072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:54.457448 22072 webserver.cc:460] Webserver started at http://127.0.0.1:45049/ using document root <none> and password file <none>
I20260927 03:57:54.457726 22072 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:54.457803 22072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:54.457927 22072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:54.459039 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "714ed9f0e55e4358ba7b009f527360bf"
format_stamp: "Formatted at 2026-09-27 03:57:54 on dist-test-slave-96fg"
server_key: "a6ccbc6d6999a785d77e2a4069cabf16"
server_key_iv: "f02141dd147e5da5b91933c5b0365d7d"
server_key_version: "encryptionkey@0"
I20260927 03:57:54.459343 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "714ed9f0e55e4358ba7b009f527360bf"
format_stamp: "Formatted at 2026-09-27 03:57:54 on dist-test-slave-96fg"
server_key: "a6ccbc6d6999a785d77e2a4069cabf16"
server_key_iv: "f02141dd147e5da5b91933c5b0365d7d"
server_key_version: "encryptionkey@0"
I20260927 03:57:54.460744 22072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:57:54.461618 22087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:54.461860 22072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:54.461992 22072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "714ed9f0e55e4358ba7b009f527360bf"
format_stamp: "Formatted at 2026-09-27 03:57:54 on dist-test-slave-96fg"
server_key: "a6ccbc6d6999a785d77e2a4069cabf16"
server_key_iv: "f02141dd147e5da5b91933c5b0365d7d"
server_key_version: "encryptionkey@0"
I20260927 03:57:54.462117 22072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:54.474207 22072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:54.477198 22072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:54.477344 22072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:54.482241 22072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43629
I20260927 03:57:54.482329 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43629 every 8 connection(s)
I20260927 03:57:54.482746 22072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:54.483909 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:54.486362 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf: Bootstrap starting.
I20260927 03:57:54.486954 22140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:54.487203 22140 log.cc:824] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:54.488077 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf: No bootstrap required, opened a new log
I20260927 03:57:54.489634 22140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } }
I20260927 03:57:54.489818 22140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:54.489854 22140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714ed9f0e55e4358ba7b009f527360bf, State: Initialized, Role: FOLLOWER
I20260927 03:57:54.489988 22140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } }
I20260927 03:57:54.490129 22140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:54.490213 22140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:54.490262 22140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:54.490806 22140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } }
I20260927 03:57:54.490898 22140 leader_election.cc:302] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 714ed9f0e55e4358ba7b009f527360bf; no voters:
I20260927 03:57:54.491083 22140 leader_election.cc:288] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:54.491180 22143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:54.491362 22140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:54.491448 22143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [term 1 LEADER]: Becoming Leader. State: Replica: 714ed9f0e55e4358ba7b009f527360bf, State: Running, Role: LEADER
I20260927 03:57:54.491607 22143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } }
I20260927 03:57:54.492309 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 714ed9f0e55e4358ba7b009f527360bf. Latest consensus state: current_term: 1 leader_uuid: "714ed9f0e55e4358ba7b009f527360bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } } }
I20260927 03:57:54.492426 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:54.492470 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "714ed9f0e55e4358ba7b009f527360bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714ed9f0e55e4358ba7b009f527360bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43629 } } }
I20260927 03:57:54.492573 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:54.492779 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22072
I20260927 03:57:54.492900 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:54.493129 1225 external_mini_cluster.cc:1428] Setting key 8ce6964743b38daffd54006a43e0953c
I20260927 03:57:54.492980 22155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:54.493861 22155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:54.495761 22155 catalog_manager.cc:1347] Generated new cluster ID: a2bc52b3149b4c95bfac66c097e3fe47
I20260927 03:57:54.495926 22155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:57:54.506327 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:54.493689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:60721 (local address 127.0.0.1:43629)
0927 03:57:54.494138 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:54.494147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:54.494189 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:57:54.494536 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:54.494549 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:54.494627 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:54.494809 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:54.495515 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:54.496976 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:54.498313 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:54.499036 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:54.499164 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:54.502121 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:54.502127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:54.502128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:54.502144 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:54.504389 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:54.504895 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:54.504898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:54.504899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:54.505020 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:54.505249 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:54.505249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:54.505250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:54.505586 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:54.505746 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:54.505893 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:57:54.506003 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":86,"threads_started":1}
I20260927 03:57:54.507592 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:54.509476 22155 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:54.510224 22155 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:54.516932 22155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 714ed9f0e55e4358ba7b009f527360bf: Generated new TSK 0
I20260927 03:57:54.517136 22155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:54.517589 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:54.509340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:51269 (local address 127.0.0.1:43629)
0927 03:57:54.509446 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:54.509448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:54.509749 (+ 301us) server_negotiation.cc:407] Connection header received
0927 03:57:54.509798 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:54.509799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:54.509812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:54.509887 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:54.510592 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:54.511402 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:54.512408 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:54.512755 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:54.512833 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:54.514349 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:54.514351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:54.514351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:54.514362 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:54.516075 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:54.516505 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:54.516507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:54.516507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:54.516577 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:54.516823 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:54.516824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:54.516824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:54.516959 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:54.517065 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:54.517451 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 03:57:54.517525 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 03:57:54.519980 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22072
W20260927 03:57:54.524209 22057 connection.cc:535] client connection to 127.0.0.1:43629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43629): Connection reset by peer (error 104)
2026-09-27T03:57:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22171](info): set up 2 sockets
Sep 27 03:57:54 dist-test-slave-96fg krb5kdc[22171](info): commencing operation
krb5kdc: starting...
W20260927 03:57:55.506328 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:55 dist-test-slave-96fg krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:55.638409 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:55.732733 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:55.733016 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:55.733050 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:55.734391 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:55.734448 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:55.734467 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:55.734484 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:55.734499 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:55.736474 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:55.736820 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:55.737095 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:55.743191 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:55.743444 22184 server_base.cc:1034] running on GCE node
W20260927 03:57:55.743191 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:55.743184 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:55.744010 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:55.744369 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:55.745616 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790481475745600 us; error 32 us; skew 500 ppm
Sep 27 03:57:55 dist-test-slave-96fg krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:55.748800 22184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:55.749362 22184 webserver.cc:460] Webserver started at http://127.0.0.1:39235/ using document root <none> and password file <none>
I20260927 03:57:55.749609 22184 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:55.749663 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:55.749796 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:55.751165 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "5ea225a4bbdf4629b6f0953168e39fd2"
format_stamp: "Formatted at 2026-09-27 03:57:55 on dist-test-slave-96fg"
server_key: "f9cb1feccc28f7d8f4722b9e2cd72426"
server_key_iv: "1152f15f00d72b5a7098958661024e9b"
server_key_version: "encryptionkey@0"
I20260927 03:57:55.751547 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "5ea225a4bbdf4629b6f0953168e39fd2"
format_stamp: "Formatted at 2026-09-27 03:57:55 on dist-test-slave-96fg"
server_key: "f9cb1feccc28f7d8f4722b9e2cd72426"
server_key_iv: "1152f15f00d72b5a7098958661024e9b"
server_key_version: "encryptionkey@0"
I20260927 03:57:55.753156 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:55.754244 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:55.754706 22184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:55.754828 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "5ea225a4bbdf4629b6f0953168e39fd2"
format_stamp: "Formatted at 2026-09-27 03:57:55 on dist-test-slave-96fg"
server_key: "f9cb1feccc28f7d8f4722b9e2cd72426"
server_key_iv: "1152f15f00d72b5a7098958661024e9b"
server_key_version: "encryptionkey@0"
I20260927 03:57:55.754906 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:55.778333 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:55.781467 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:55.781643 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:55.789403 22184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38265
I20260927 03:57:55.789486 22251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38265 every 8 connection(s)
I20260927 03:57:55.790074 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:55.791472 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:55.794507 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22184
I20260927 03:57:55.794638 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:55.794701 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: Bootstrap starting.
I20260927 03:57:55.794837 1225 external_mini_cluster.cc:1428] Setting key d3e135c6e602ddf2de5801b406fd0e0c
I20260927 03:57:55.795429 22252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:55.795871 22252 log.cc:824] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:55.796547 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: No bootstrap required, opened a new log
I20260927 03:57:55.798532 22252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } }
I20260927 03:57:55.798702 22252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:55.798727 22252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea225a4bbdf4629b6f0953168e39fd2, State: Initialized, Role: FOLLOWER
I20260927 03:57:55.798851 22252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } }
I20260927 03:57:55.798928 22252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:55.798951 22252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:55.799000 22252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:55.799590 22252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } }
I20260927 03:57:55.799695 22252 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea225a4bbdf4629b6f0953168e39fd2; no voters:
I20260927 03:57:55.799873 22252 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:55.799916 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:55.800099 22257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea225a4bbdf4629b6f0953168e39fd2, State: Running, Role: LEADER
I20260927 03:57:55.800168 22252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:55.801154 22257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } }
I20260927 03:57:55.801970 22259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } } }
I20260927 03:57:55.802650 22259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:55.802203 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea225a4bbdf4629b6f0953168e39fd2. Latest consensus state: current_term: 1 leader_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea225a4bbdf4629b6f0953168e39fd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38265 } } }
I20260927 03:57:55.802302 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:55.795774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:43009 (local address 127.0.0.1:38265)
0927 03:57:55.796040 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:55.796053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:55.796102 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:57:55.796746 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:55.796755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:55.796821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:55.796987 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:55.797234 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:55.798326 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:55.799333 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:55.800060 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:55.800192 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:55.801223 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:55.801228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:55.801229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:55.801499 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:55.801552 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:55.801571 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:55.801653 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:55.802006 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 03:57:55.802075 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260927 03:57:55.803004 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2 [sys.catalog]: This master's current role is: LEADER
W20260927 03:57:55.803345 22271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:55.803392 22271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:55.803447 22273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:55.803625 22273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:55.804401 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:55.806923 22273 catalog_manager.cc:1347] Generated new cluster ID: c75219ffabd743bf9a170a29d543c4e1
I20260927 03:57:55.806980 22273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:55.809998 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:55.806057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:35921 (local address 127.0.0.1:38265)
0927 03:57:55.806099 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:55.806101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:55.806281 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:57:55.806349 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:55.806349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:55.806362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:55.806424 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:55.806613 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:55.807399 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:55.808497 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:55.808829 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:55.808905 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:55.809705 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:55.809707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:55.809707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:55.809713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:55.809737 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:55.809742 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:55.809821 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:55.809892 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:57:55.809906 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:57:55.829470 22273 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:55.830400 22273 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:55.836663 22273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea225a4bbdf4629b6f0953168e39fd2: Generated new TSK 0
I20260927 03:57:55.836907 22273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:55.845973 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22184
W20260927 03:57:55.851073 22169 connection.cc:535] client connection to 127.0.0.1:38265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38265): Connection reset by peer (error 104)
2026-09-27T03:57:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:55 dist-test-slave-96fg krb5kdc[22283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:55 dist-test-slave-96fg krb5kdc[22283](info): set up 2 sockets
Sep 27 03:57:55 dist-test-slave-96fg krb5kdc[22283](info): commencing operation
krb5kdc: starting...
W20260927 03:57:56.834146 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:56 dist-test-slave-96fg krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:56.974675 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:57.071846 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:57.072142 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:57.072170 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:57.073511 22296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:57.073562 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:57.073575 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:57.073587 22296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:57.073598 22296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:57.075300 22296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:57.075583 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:57.075825 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:57.082096 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:57.082260 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:57.082049 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:57.082438 22296 server_base.cc:1034] running on GCE node
I20260927 03:57:57.082769 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:57.083220 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:57.084427 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790481477084414 us; error 37 us; skew 500 ppm
Sep 27 03:57:57 dist-test-slave-96fg krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:57.087785 22296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:57.088368 22296 webserver.cc:460] Webserver started at http://127.0.0.1:34137/ using document root <none> and password file <none>
I20260927 03:57:57.088629 22296 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:57.088683 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:57.089025 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:57.090469 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "944a1404bc4b4f0c9ad3030bb8434aae"
format_stamp: "Formatted at 2026-09-27 03:57:57 on dist-test-slave-96fg"
server_key: "d3682f30638e1d9792859c814861fd6e"
server_key_iv: "62be0adca7bd4da5caad0bd66cefb5e3"
server_key_version: "encryptionkey@0"
I20260927 03:57:57.090858 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "944a1404bc4b4f0c9ad3030bb8434aae"
format_stamp: "Formatted at 2026-09-27 03:57:57 on dist-test-slave-96fg"
server_key: "d3682f30638e1d9792859c814861fd6e"
server_key_iv: "62be0adca7bd4da5caad0bd66cefb5e3"
server_key_version: "encryptionkey@0"
I20260927 03:57:57.092542 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:57:57.093762 22311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:57.094012 22296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:57.094177 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "944a1404bc4b4f0c9ad3030bb8434aae"
format_stamp: "Formatted at 2026-09-27 03:57:57 on dist-test-slave-96fg"
server_key: "d3682f30638e1d9792859c814861fd6e"
server_key_iv: "62be0adca7bd4da5caad0bd66cefb5e3"
server_key_version: "encryptionkey@0"
I20260927 03:57:57.094255 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:57.098227 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:57.100618 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:57.100788 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:57.106309 22296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34685
I20260927 03:57:57.106353 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34685 every 8 connection(s)
I20260927 03:57:57.106750 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:57.107879 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:57.110411 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: Bootstrap starting.
I20260927 03:57:57.110772 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22296
I20260927 03:57:57.110873 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:57.111079 1225 external_mini_cluster.cc:1428] Setting key f942051a49a437bdb8afb6ab624bd744
I20260927 03:57:57.111086 22364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:57.111356 22364 log.cc:824] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:57.112471 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: No bootstrap required, opened a new log
I20260927 03:57:57.114470 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:57.111742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:52803 (local address 127.0.0.1:34685)
0927 03:57:57.111957 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:57.111969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:57.112016 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:57:57.112794 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:57.112805 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:57.112843 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:57.113054 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:57.113854 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:57.113858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:57.113858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:57.113863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:57.113923 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:57.113941 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:57.114090 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:57.114177 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:57:57.114232 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 03:57:57.115139 22364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 03:57:57.115330 22364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:57.115370 22364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944a1404bc4b4f0c9ad3030bb8434aae, State: Initialized, Role: FOLLOWER
I20260927 03:57:57.115520 22364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 03:57:57.115629 22364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:57.115677 22364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:57.115729 22364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:57.115754 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:57.116650 22364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 03:57:57.116770 22364 leader_election.cc:302] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944a1404bc4b4f0c9ad3030bb8434aae; no voters:
I20260927 03:57:57.116998 22364 leader_election.cc:288] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:57.117072 22370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:57.117316 22364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:57.117400 22370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [term 1 LEADER]: Becoming Leader. State: Replica: 944a1404bc4b4f0c9ad3030bb8434aae, State: Running, Role: LEADER
I20260927 03:57:57.117575 22370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260927 03:57:57.118005 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260927 03:57:57.118127 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:57.118300 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944a1404bc4b4f0c9ad3030bb8434aae. Latest consensus state: current_term: 1 leader_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944a1404bc4b4f0c9ad3030bb8434aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260927 03:57:57.118820 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:57.119112 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:57.117527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:44111 (local address 127.0.0.1:34685)
0927 03:57:57.117565 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:57.117567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:57.117848 (+ 281us) server_negotiation.cc:407] Connection header received
0927 03:57:57.117952 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:57.117953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:57.117959 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:57.117992 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:57.118871 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:57.118872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:57.118873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:57.118877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:57.118894 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:57.118899 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:57.118941 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:57.119028 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:57:57.119044 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:57:57.119474 22390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:57:57.119544 22390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:57:57.119614 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:57.119843 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:57.121702 22383 catalog_manager.cc:1347] Generated new cluster ID: 110c298409384a01a0423fc617d6b56d
I20260927 03:57:57.121764 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:57.159742 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:57.160039 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:57.169924 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 944a1404bc4b4f0c9ad3030bb8434aae: Generated new TSK 0
I20260927 03:57:57.170296 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:57.181663 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22296
2026-09-27T03:57:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:57 dist-test-slave-96fg krb5kdc[22395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:57 dist-test-slave-96fg krb5kdc[22395](info): set up 2 sockets
Sep 27 03:57:57 dist-test-slave-96fg krb5kdc[22395](info): commencing operation
krb5kdc: starting...
W20260927 03:57:58.185952 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:58.318570 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:58.403797 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:58.404016 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:58.404039 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:58.405228 22408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:58.405365 22408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:58.405449 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:58.405489 22408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:58.405524 22408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:58.407326 22408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:58.407593 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:58.407836 22408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:58.413097 22413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:58.413326 22408 server_base.cc:1034] running on GCE node
W20260927 03:57:58.413086 22414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:58.413091 22416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:58.413725 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:58.414130 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:58.415330 22408 hybrid_clock.cc:648] HybridClock initialized: now 1790481478415304 us; error 39 us; skew 500 ppm
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:58.422420 22408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:58.423007 22408 webserver.cc:460] Webserver started at http://127.0.0.1:40113/ using document root <none> and password file <none>
I20260927 03:57:58.423280 22408 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:58.423336 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:58.423517 22408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:58.424726 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "1e920cc9939f44fb930b8486b29a617a"
format_stamp: "Formatted at 2026-09-27 03:57:58 on dist-test-slave-96fg"
server_key: "c1e65feb163a962d8c0111e93ad80476"
server_key_iv: "029d2fd2f475a56499883ef307736b2f"
server_key_version: "encryptionkey@0"
I20260927 03:57:58.425100 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "1e920cc9939f44fb930b8486b29a617a"
format_stamp: "Formatted at 2026-09-27 03:57:58 on dist-test-slave-96fg"
server_key: "c1e65feb163a962d8c0111e93ad80476"
server_key_iv: "029d2fd2f475a56499883ef307736b2f"
server_key_version: "encryptionkey@0"
I20260927 03:57:58.426651 22408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:57:58.427696 22423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:58.428025 22408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:57:58.428135 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "1e920cc9939f44fb930b8486b29a617a"
format_stamp: "Formatted at 2026-09-27 03:57:58 on dist-test-slave-96fg"
server_key: "c1e65feb163a962d8c0111e93ad80476"
server_key_iv: "029d2fd2f475a56499883ef307736b2f"
server_key_version: "encryptionkey@0"
I20260927 03:57:58.428212 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:58.440397 22408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:58.443235 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:58.443403 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:58.448447 22408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40855
I20260927 03:57:58.448511 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40855 every 8 connection(s)
I20260927 03:57:58.448905 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:58.450117 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:58.452574 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a: Bootstrap starting.
I20260927 03:57:58.453164 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:58.453436 22476 log.cc:824] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:58.453733 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22408
I20260927 03:57:58.453874 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:58.454105 1225 external_mini_cluster.cc:1428] Setting key ebcc75c13c10bc07a62b3bc310f22e5c
I20260927 03:57:58.454406 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a: No bootstrap required, opened a new log
I20260927 03:57:58.456598 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260927 03:57:58.456758 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:58.456781 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e920cc9939f44fb930b8486b29a617a, State: Initialized, Role: FOLLOWER
I20260927 03:57:58.456909 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260927 03:57:58.456984 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:58.457015 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:58.457065 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:58.457672 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260927 03:57:58.457763 22476 leader_election.cc:302] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e920cc9939f44fb930b8486b29a617a; no voters:
I20260927 03:57:58.457944 22476 leader_election.cc:288] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:58.457985 22481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:58.458129 22481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [term 1 LEADER]: Becoming Leader. State: Replica: 1e920cc9939f44fb930b8486b29a617a, State: Running, Role: LEADER
I20260927 03:57:58.458253 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:58.458312 22481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260927 03:57:58.458666 22481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e920cc9939f44fb930b8486b29a617a. Latest consensus state: current_term: 1 leader_uuid: "1e920cc9939f44fb930b8486b29a617a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260927 03:57:58.458835 22481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:58.459096 22489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:58.459664 22489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:58.460553 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e920cc9939f44fb930b8486b29a617a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e920cc9939f44fb930b8486b29a617a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260927 03:57:58.460736 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:58.462244 22489 catalog_manager.cc:1347] Generated new cluster ID: 584ffc8c09fd4bdf8793c96ec93e3425
I20260927 03:57:58.462314 22489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:57:58.466836 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:58.454761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:45919 (local address 127.0.0.1:40855)
0927 03:57:58.454995 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:58.455006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:58.455039 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:57:58.455772 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:58.455780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:58.455844 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:58.455999 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:58.456681 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:58.458215 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:58.459254 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:58.459913 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:58.460034 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:58.462949 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:58.462955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:58.462955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:58.462971 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:58.465022 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:58.465517 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:58.465520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:58.465521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:58.465609 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:58.465815 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:58.465816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:58.465816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:58.466117 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:58.466261 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:58.466521 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:57:58.466623 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260927 03:57:58.467870 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:58.479017 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:58.470467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:45731 (local address 127.0.0.1:40855)
0927 03:57:58.470509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:58.470511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:58.473190 (+ 2679us) server_negotiation.cc:407] Connection header received
0927 03:57:58.473306 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:58.473307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:58.473320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:58.473369 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:58.473932 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:58.474507 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:58.475226 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:58.475477 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:58.475533 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:58.476700 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:58.476702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:58.476702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:57:58.476709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:58.477930 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:58.478297 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:58.478298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:58.478299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:58.478345 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:57:58.478532 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:57:58.478532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:57:58.478533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:57:58.478633 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:58.478731 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:58.478865 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:57:58.478926 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:57:58.483054 22489 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:58.483764 22489 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:58.488287 22489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e920cc9939f44fb930b8486b29a617a: Generated new TSK 0
I20260927 03:57:58.488461 22489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:58.491859 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22408
W20260927 03:57:58.495743 22393 connection.cc:535] client connection to 127.0.0.1:40855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40855): Connection reset by peer (error 104)
2026-09-27T03:57:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22507](info): set up 2 sockets
Sep 27 03:57:58 dist-test-slave-96fg krb5kdc[22507](info): commencing operation
krb5kdc: starting...
W20260927 03:57:59.446105 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:57:59 dist-test-slave-96fg krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:57:59.575047 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:57:59.661902 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:57:59.662211 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:57:59.662241 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:57:59.663543 22520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:57:59.663621 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:57:59.663638 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:57:59.663653 22520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:57:59.663666 22520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:57:59.665683 22520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:57:59.666296 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:57:59.666589 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:57:59.672544 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:57:59.672631 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:59.672745 22520 server_base.cc:1034] running on GCE node
W20260927 03:57:59.673097 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:57:59.673473 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:57:59.673910 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:57:59.675184 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790481479675145 us; error 62 us; skew 500 ppm
Sep 27 03:57:59 dist-test-slave-96fg krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:57:59.678752 22520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:57:59.679337 22520 webserver.cc:460] Webserver started at http://127.0.0.1:44225/ using document root <none> and password file <none>
I20260927 03:57:59.679548 22520 fs_manager.cc:359] Metadata directory not provided
I20260927 03:57:59.679598 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:57:59.679716 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:57:59.681105 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "02010903579040518c876fb8c85c4db4"
format_stamp: "Formatted at 2026-09-27 03:57:59 on dist-test-slave-96fg"
server_key: "9d5bd29e92bae165c35d7a84269cdeee"
server_key_iv: "df7fd832763dd13a4a5ecb65c438f3fc"
server_key_version: "encryptionkey@0"
I20260927 03:57:59.681452 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "02010903579040518c876fb8c85c4db4"
format_stamp: "Formatted at 2026-09-27 03:57:59 on dist-test-slave-96fg"
server_key: "9d5bd29e92bae165c35d7a84269cdeee"
server_key_iv: "df7fd832763dd13a4a5ecb65c438f3fc"
server_key_version: "encryptionkey@0"
I20260927 03:57:59.683148 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 03:57:59.684230 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:57:59.684657 22520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:57:59.684780 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "02010903579040518c876fb8c85c4db4"
format_stamp: "Formatted at 2026-09-27 03:57:59 on dist-test-slave-96fg"
server_key: "9d5bd29e92bae165c35d7a84269cdeee"
server_key_iv: "df7fd832763dd13a4a5ecb65c438f3fc"
server_key_version: "encryptionkey@0"
I20260927 03:57:59.684913 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:57:59.693574 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:57:59.695991 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:57:59.696143 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:57:59.700781 22520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33905
I20260927 03:57:59.700879 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33905 every 8 connection(s)
I20260927 03:57:59.701226 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:57:59.702404 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:57:59.705464 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4: Bootstrap starting.
I20260927 03:57:59.706219 22588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4: Neither blocks nor log segments found. Creating new log.
I20260927 03:57:59.706529 22588 log.cc:824] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:57:59.707597 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4: No bootstrap required, opened a new log
I20260927 03:57:59.709735 22588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260927 03:57:59.709990 22588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:57:59.710089 22588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02010903579040518c876fb8c85c4db4, State: Initialized, Role: FOLLOWER
I20260927 03:57:59.710268 22588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260927 03:57:59.710348 22588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:57:59.710384 22588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:57:59.710438 22588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:57:59.711148 22588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260927 03:57:59.711262 22588 leader_election.cc:302] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02010903579040518c876fb8c85c4db4; no voters:
I20260927 03:57:59.711352 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22520
I20260927 03:57:59.711480 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:57:59.711481 22588 leader_election.cc:288] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:57:59.711580 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:57:59.711683 1225 external_mini_cluster.cc:1428] Setting key b771f8b4b890cb4fe97750ae0cb6f4c4
I20260927 03:57:59.711807 22588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:57:59.711951 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [term 1 LEADER]: Becoming Leader. State: Replica: 02010903579040518c876fb8c85c4db4, State: Running, Role: LEADER
I20260927 03:57:59.712102 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260927 03:57:59.712651 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02010903579040518c876fb8c85c4db4. Latest consensus state: current_term: 1 leader_uuid: "02010903579040518c876fb8c85c4db4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260927 03:57:59.712766 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:59.712821 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02010903579040518c876fb8c85c4db4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02010903579040518c876fb8c85c4db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260927 03:57:59.712913 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:57:59.713007 22602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:57:59.713716 22602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:57:59.715693 22602 catalog_manager.cc:1347] Generated new cluster ID: 7a83c8bd91394714bd8910b907f9c588
I20260927 03:57:59.715754 22602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:57:59.718698 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:59.712248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:53209 (local address 127.0.0.1:33905)
0927 03:57:59.712467 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:59.712480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:59.712731 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:57:59.712983 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:59.712992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:59.713062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:59.713182 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:59.713656 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:59.714970 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:59.716004 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:59.716691 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:59.716867 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:59.717611 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:59.717616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:59.717616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:59.717882 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:59.717934 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:59.717954 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:59.718109 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:59.718227 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:57:59.718267 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260927 03:57:59.720060 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:57:59.725548 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:57:59.721600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:49317 (local address 127.0.0.1:33905)
0927 03:57:59.721645 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:57:59.721647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:57:59.721661 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:57:59.721740 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:57:59.721741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:57:59.721754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:57:59.721848 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:57:59.722000 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:59.722897 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:59.723922 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:57:59.724279 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:57:59.724365 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:57:59.725197 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:57:59.725199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:57:59.725200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:57:59.725205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:57:59.725229 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:57:59.725235 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:57:59.725306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:57:59.725398 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:57:59.725416 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:57:59.757221 22602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:57:59.758149 22602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:57:59.771876 22602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02010903579040518c876fb8c85c4db4: Generated new TSK 0
I20260927 03:57:59.772164 22602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:57:59.784646 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22520
W20260927 03:57:59.788967 22505 connection.cc:535] client connection to 127.0.0.1:33905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33905): Connection reset by peer (error 104)
2026-09-27T03:57:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:57:59 dist-test-slave-96fg krb5kdc[22619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:57:59 dist-test-slave-96fg krb5kdc[22619](info): set up 2 sockets
Sep 27 03:57:59 dist-test-slave-96fg krb5kdc[22619](info): commencing operation
krb5kdc: starting...
W20260927 03:58:00.755092 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:00 dist-test-slave-96fg krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:58:00.898836 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:00.992440 22632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:00.992717 22632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:00.992743 22632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:00.994127 22632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:00.994184 22632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:00.994197 22632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:00.994210 22632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:00.994223 22632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:00.995930 22632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:00.996214 22632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:00.996464 22632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:01.002307 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:01.002357 22640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:01.002835 22632 server_base.cc:1034] running on GCE node
W20260927 03:58:01.002302 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:01.003321 22632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:01.003710 22632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:01.004916 22632 hybrid_clock.cc:648] HybridClock initialized: now 1790481481004885 us; error 49 us; skew 500 ppm
Sep 27 03:58:01 dist-test-slave-96fg krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:01.008328 22632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:58:01.008869 22632 webserver.cc:460] Webserver started at http://127.0.0.1:41145/ using document root <none> and password file <none>
I20260927 03:58:01.009132 22632 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:01.009187 22632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:01.009313 22632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:01.010587 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "5892f858cd18451fa985cbdad4f24430"
format_stamp: "Formatted at 2026-09-27 03:58:01 on dist-test-slave-96fg"
server_key: "9be5cf44855f1da5368e39f2f06bb7d9"
server_key_iv: "d68cfb2f79aebd89dd6a34262056fb2c"
server_key_version: "encryptionkey@0"
I20260927 03:58:01.011006 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "5892f858cd18451fa985cbdad4f24430"
format_stamp: "Formatted at 2026-09-27 03:58:01 on dist-test-slave-96fg"
server_key: "9be5cf44855f1da5368e39f2f06bb7d9"
server_key_iv: "d68cfb2f79aebd89dd6a34262056fb2c"
server_key_version: "encryptionkey@0"
I20260927 03:58:01.012553 22632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:01.013537 22647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:01.013818 22632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:01.013926 22632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "5892f858cd18451fa985cbdad4f24430"
format_stamp: "Formatted at 2026-09-27 03:58:01 on dist-test-slave-96fg"
server_key: "9be5cf44855f1da5368e39f2f06bb7d9"
server_key_iv: "d68cfb2f79aebd89dd6a34262056fb2c"
server_key_version: "encryptionkey@0"
I20260927 03:58:01.014062 22632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:01.017627 22632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:01.019806 22632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:01.019929 22632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:01.024327 22632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42419
I20260927 03:58:01.024392 22699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42419 every 8 connection(s)
I20260927 03:58:01.024742 22632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:01.025059 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22632
I20260927 03:58:01.025190 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:01.025450 1225 external_mini_cluster.cc:1428] Setting key b1cfe56eaf75378f1ca413d8da419df3
I20260927 03:58:01.026183 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:01.028873 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: Bootstrap starting.
I20260927 03:58:01.029516 22700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:01.029809 22700 log.cc:824] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:01.030532 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: No bootstrap required, opened a new log
I20260927 03:58:01.031626 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:01.026177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:44985 (local address 127.0.0.1:42419)
0927 03:58:01.029233 (+ 3056us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:01.029246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:01.029287 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:58:01.030004 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:01.030013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:01.030092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:01.030269 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:01.031120 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:01.031124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:01.031124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:01.031129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:01.031175 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:01.031187 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:01.031264 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:01.031375 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:58:01.031425 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2973,"thread_start_us":85,"threads_started":1}
I20260927 03:58:01.032836 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:01.033033 22700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260927 03:58:01.033221 22700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:01.033303 22700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5892f858cd18451fa985cbdad4f24430, State: Initialized, Role: FOLLOWER
I20260927 03:58:01.033455 22700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260927 03:58:01.033550 22700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:01.033591 22700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:01.033640 22700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:01.034577 22700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260927 03:58:01.034709 22700 leader_election.cc:302] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5892f858cd18451fa985cbdad4f24430; no voters:
I20260927 03:58:01.034945 22700 leader_election.cc:288] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:01.034981 22710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:01.035147 22710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [term 1 LEADER]: Becoming Leader. State: Replica: 5892f858cd18451fa985cbdad4f24430, State: Running, Role: LEADER
I20260927 03:58:01.035293 22700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:01.035293 22710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260927 03:58:01.035816 22710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5892f858cd18451fa985cbdad4f24430. Latest consensus state: current_term: 1 leader_uuid: "5892f858cd18451fa985cbdad4f24430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260927 03:58:01.036116 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:01.034746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:55455 (local address 127.0.0.1:42419)
0927 03:58:01.034806 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:01.034808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:01.034935 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:58:01.034989 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:01.034990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:01.034996 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:01.035040 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:01.035842 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:01.035844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:01.035844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:01.035849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:01.035865 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:01.035869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:01.035898 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:01.035997 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:58:01.036011 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:58:01.036213 22710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:01.036013 22711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5892f858cd18451fa985cbdad4f24430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5892f858cd18451fa985cbdad4f24430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260927 03:58:01.036754 22711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430 [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:01.037307 22726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:01.037355 22726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:01.037566 22719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:01.037740 22719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:01.039781 22719 catalog_manager.cc:1347] Generated new cluster ID: 405461785b7642888c3b0dd1d85f2f49
I20260927 03:58:01.039839 22719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:01.047010 22719 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:01.047295 22719 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:01.052654 22719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5892f858cd18451fa985cbdad4f24430: Generated new TSK 0
I20260927 03:58:01.052922 22719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:01.059405 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22632
2026-09-27T03:58:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:01 dist-test-slave-96fg krb5kdc[22731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:01 dist-test-slave-96fg krb5kdc[22731](info): set up 2 sockets
Sep 27 03:58:01 dist-test-slave-96fg krb5kdc[22731](info): commencing operation
krb5kdc: starting...
W20260927 03:58:02.062467 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:02.199172 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40135
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:02.297875 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:02.298167 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:02.298198 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:02.299521 22744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:02.299583 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:02.299597 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:02.299609 22744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:02.299620 22744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:02.301282 22744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:02.301628 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:02.301906 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:02.308696 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:02.308696 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:02.308781 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:02.308863 22744 server_base.cc:1034] running on GCE node
I20260927 03:58:02.309374 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:02.309756 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:02.311048 22744 hybrid_clock.cc:648] HybridClock initialized: now 1790481482310994 us; error 74 us; skew 500 ppm
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:02.314600 22744 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:02.315328 22744 webserver.cc:460] Webserver started at http://127.1.50.126:36599/ using document root <none> and password file <none>
I20260927 03:58:02.315639 22744 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:02.315696 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:02.315824 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:02.317104 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "e372cf00b8e64fcd91084ab78220a52f"
format_stamp: "Formatted at 2026-09-27 03:58:02 on dist-test-slave-96fg"
server_key: "2726bf1707a447b0808b140b48179462"
server_key_iv: "812340beb98511afc7017042049ff945"
server_key_version: "encryptionkey@0"
I20260927 03:58:02.317487 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "e372cf00b8e64fcd91084ab78220a52f"
format_stamp: "Formatted at 2026-09-27 03:58:02 on dist-test-slave-96fg"
server_key: "2726bf1707a447b0808b140b48179462"
server_key_iv: "812340beb98511afc7017042049ff945"
server_key_version: "encryptionkey@0"
I20260927 03:58:02.319190 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 03:58:02.320394 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:02.320716 22744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:02.320854 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "e372cf00b8e64fcd91084ab78220a52f"
format_stamp: "Formatted at 2026-09-27 03:58:02 on dist-test-slave-96fg"
server_key: "2726bf1707a447b0808b140b48179462"
server_key_iv: "812340beb98511afc7017042049ff945"
server_key_version: "encryptionkey@0"
I20260927 03:58:02.320955 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:02.355481 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:02.357766 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:02.357906 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:02.362322 22744 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:40135
I20260927 03:58:02.362425 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:40135 every 8 connection(s)
I20260927 03:58:02.362756 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:02.363880 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:02.366897 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f: Bootstrap starting.
I20260927 03:58:02.367169 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22744
I20260927 03:58:02.367295 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:02.367527 1225 external_mini_cluster.cc:1428] Setting key 0d0c953d2d8e6d9aaaa13e21623dbe48
I20260927 03:58:02.367914 22812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:02.368369 22812 log.cc:824] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:02.369786 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f: No bootstrap required, opened a new log
I20260927 03:58:02.372584 22812 raft_consensus.cc:348] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } }
I20260927 03:58:02.372828 22812 raft_consensus.cc:374] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:02.372871 22812 raft_consensus.cc:729] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e372cf00b8e64fcd91084ab78220a52f, State: Initialized, Role: FOLLOWER
I20260927 03:58:02.373054 22812 consensus_queue.cc:260] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } }
I20260927 03:58:02.373147 22812 raft_consensus.cc:388] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:02.373183 22812 raft_consensus.cc:482] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:02.373247 22812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:02.374109 22812 raft_consensus.cc:504] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } }
I20260927 03:58:02.374235 22812 leader_election.cc:302] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e372cf00b8e64fcd91084ab78220a52f; no voters:
I20260927 03:58:02.374454 22812 leader_election.cc:288] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:02.374508 22817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:02.374634 22817 raft_consensus.cc:686] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [term 1 LEADER]: Becoming Leader. State: Replica: e372cf00b8e64fcd91084ab78220a52f, State: Running, Role: LEADER
I20260927 03:58:02.374739 22817 consensus_queue.cc:237] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } }
I20260927 03:58:02.374812 22812 sys_catalog.cc:564] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:02.375154 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e372cf00b8e64fcd91084ab78220a52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } } }
I20260927 03:58:02.375233 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:02.375378 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e372cf00b8e64fcd91084ab78220a52f. Latest consensus state: current_term: 1 leader_uuid: "e372cf00b8e64fcd91084ab78220a52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e372cf00b8e64fcd91084ab78220a52f" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40135 } } }
I20260927 03:58:02.375566 22821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:58:02.375667 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:02.376808 22821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:02.379426 22821 catalog_manager.cc:1347] Generated new cluster ID: 2043f139b9ad4312b6cac76aa4468b0e
I20260927 03:58:02.379487 22821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:02.381770 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:02.368398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47233 (local address 127.1.50.126:40135)
0927 03:58:02.369004 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:02.369018 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:02.369079 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:58:02.369791 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:02.369802 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:02.369872 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:02.370074 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:02.370720 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:02.371675 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:02.372787 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:02.373553 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:02.373686 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:02.377136 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:02.377141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:02.377142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:02.377158 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:02.379211 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:02.379940 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:02.379944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:02.379944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:02.380065 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:02.380361 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:02.380362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:02.380362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:02.380674 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:02.380859 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:02.381422 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 03:58:02.381525 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":258,"threads_started":1}
I20260927 03:58:02.382695 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:02.393010 22821 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:02.393728 22821 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:02.398558 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:02.386499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51053 (local address 127.1.50.126:40135)
0927 03:58:02.386583 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:02.386587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:02.390366 (+ 3779us) server_negotiation.cc:407] Connection header received
0927 03:58:02.390467 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:02.390468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:02.390483 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:02.390547 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:02.391395 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:02.392170 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:02.393319 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:02.393667 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:02.393760 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:02.395481 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:02.395483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:02.395484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:02.395493 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:02.397167 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:02.397666 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:02.397666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:02.397667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:02.397735 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:02.397963 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:02.397963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:02.397964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:02.398121 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:02.398227 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:02.398400 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 03:58:02.398475 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:58:02.399242 22821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e372cf00b8e64fcd91084ab78220a52f: Generated new TSK 0
I20260927 03:58:02.399498 22821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:02.407231 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22744
W20260927 03:58:02.411813 22729 connection.cc:535] client connection to 127.1.50.126:40135 recv error: Network error: failed to read from TLS socket (remote: 127.1.50.126:40135): Connection reset by peer (error 104)
2026-09-27T03:58:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22843](info): set up 2 sockets
Sep 27 03:58:02 dist-test-slave-96fg krb5kdc[22843](info): commencing operation
krb5kdc: starting...
W20260927 03:58:03.394426 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:03 dist-test-slave-96fg krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:03.526196 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:33405
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:03.608943 22856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:03.609207 22856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:03.609246 22856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:03.610617 22856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:03.610669 22856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:03.610689 22856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:03.610705 22856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:03.610720 22856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:03.612346 22856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:33405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:03.612635 22856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:03.612864 22856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:03.618353 22861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:03.618340 22864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:03.618330 22862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:03.618705 22856 server_base.cc:1034] running on GCE node
I20260927 03:58:03.618976 22856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:03.619366 22856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:03.620549 22856 hybrid_clock.cc:648] HybridClock initialized: now 1790481483620510 us; error 63 us; skew 500 ppm
Sep 27 03:58:03 dist-test-slave-96fg krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:03.627732 22856 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:03.628342 22856 webserver.cc:460] Webserver started at http://127.1.50.126:34783/ using document root <none> and password file <none>
I20260927 03:58:03.628647 22856 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:03.628705 22856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:03.628806 22856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:03.629889 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "9444332f2f6944ba81c20d80744e873c"
format_stamp: "Formatted at 2026-09-27 03:58:03 on dist-test-slave-96fg"
server_key: "33296c9b1c9039f05d480a86d9ba084e"
server_key_iv: "0337ced563c59bf9fd30a8fec1aa2bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:03.630273 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "9444332f2f6944ba81c20d80744e873c"
format_stamp: "Formatted at 2026-09-27 03:58:03 on dist-test-slave-96fg"
server_key: "33296c9b1c9039f05d480a86d9ba084e"
server_key_iv: "0337ced563c59bf9fd30a8fec1aa2bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:03.631769 22856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:03.632781 22871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:03.633121 22856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:03.633245 22856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "9444332f2f6944ba81c20d80744e873c"
format_stamp: "Formatted at 2026-09-27 03:58:03 on dist-test-slave-96fg"
server_key: "33296c9b1c9039f05d480a86d9ba084e"
server_key_iv: "0337ced563c59bf9fd30a8fec1aa2bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:03.633342 22856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:03.684568 22856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:03.686903 22856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:03.687054 22856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:03.691363 22856 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:33405
I20260927 03:58:03.691439 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:33405 every 8 connection(s)
I20260927 03:58:03.691767 22856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:03.692800 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22856
I20260927 03:58:03.692925 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:03.693013 22924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:03.693176 1225 external_mini_cluster.cc:1428] Setting key 190346b136ba13da776220acf3902264
I20260927 03:58:03.697983 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c: Bootstrap starting.
I20260927 03:58:03.698873 22924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:03.699240 22924 log.cc:824] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:03.700341 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c: No bootstrap required, opened a new log
I20260927 03:58:03.702217 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:03.693866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36413 (local address 127.1.50.126:33405)
0927 03:58:03.694157 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:03.694169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:03.694209 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:58:03.695052 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:03.695065 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:03.695151 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:03.695451 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:03.695935 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:03.697511 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:03.698625 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:03.699293 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:03.699427 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:03.700487 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:03.700495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:03.700495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:03.700745 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:03.700794 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:03.700811 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:03.700924 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:03.701613 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 03:58:03.701655 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":161,"threads_started":1}
I20260927 03:58:03.702467 22924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } }
I20260927 03:58:03.702667 22924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:03.702701 22924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9444332f2f6944ba81c20d80744e873c, State: Initialized, Role: FOLLOWER
I20260927 03:58:03.702849 22924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } }
I20260927 03:58:03.702934 22924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:03.702963 22924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:03.702993 22924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:03.703817 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:03.703843 22924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } }
I20260927 03:58:03.703946 22924 leader_election.cc:302] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9444332f2f6944ba81c20d80744e873c; no voters:
I20260927 03:58:03.704142 22924 leader_election.cc:288] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:03.704226 22929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:03.704432 22924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:03.704491 22929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [term 1 LEADER]: Becoming Leader. State: Replica: 9444332f2f6944ba81c20d80744e873c, State: Running, Role: LEADER
I20260927 03:58:03.704687 22929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } }
I20260927 03:58:03.705135 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9444332f2f6944ba81c20d80744e873c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } } }
I20260927 03:58:03.705226 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:03.705118 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9444332f2f6944ba81c20d80744e873c. Latest consensus state: current_term: 1 leader_uuid: "9444332f2f6944ba81c20d80744e873c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444332f2f6944ba81c20d80744e873c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 33405 } } }
I20260927 03:58:03.705492 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:03.705693 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:03.706323 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:03.708148 22942 catalog_manager.cc:1347] Generated new cluster ID: 6a8eb46eae35447f86e283fe25f15cd9
I20260927 03:58:03.708207 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:03.709424 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:03.706146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52695 (local address 127.1.50.126:33405)
0927 03:58:03.706192 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:03.706194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:03.706383 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:58:03.706488 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:03.706489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:03.706500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:03.706529 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:03.706767 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:03.707315 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:03.708172 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:03.708427 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:03.708485 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:03.709192 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:03.709193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:03.709194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:03.709199 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:03.709215 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:03.709220 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:03.709264 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:03.709330 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 03:58:03.709341 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:58:03.722091 22942 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:03.722994 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:03.731031 22942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9444332f2f6944ba81c20d80744e873c: Generated new TSK 0
I20260927 03:58:03.731307 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:03.735342 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22856
W20260927 03:58:03.739857 22841 connection.cc:535] client connection to 127.1.50.126:33405 recv error: Network error: failed to read from TLS socket (remote: 127.1.50.126:33405): Connection reset by peer (error 104)
2026-09-27T03:58:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:03 dist-test-slave-96fg krb5kdc[22955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:03 dist-test-slave-96fg krb5kdc[22955](info): set up 2 sockets
Sep 27 03:58:03 dist-test-slave-96fg krb5kdc[22955](info): commencing operation
krb5kdc: starting...
W20260927 03:58:04.706212 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:04 dist-test-slave-96fg krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:04.837458 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39061
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:04.934240 22968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:04.934547 22968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:04.934589 22968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:04.936118 22968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:04.936184 22968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:04.936205 22968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:04.936221 22968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:04.936237 22968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:04.938151 22968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:04.938475 22968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:04.938737 22968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:04.944914 22973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:04.944916 22976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:04.945135 22968 server_base.cc:1034] running on GCE node
W20260927 03:58:04.944959 22974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:04.945629 22968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:04.945974 22968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:04.947182 22968 hybrid_clock.cc:648] HybridClock initialized: now 1790481484947135 us; error 65 us; skew 500 ppm
Sep 27 03:58:04 dist-test-slave-96fg krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:04.950189 22968 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:04.950713 22968 webserver.cc:460] Webserver started at http://127.1.50.126:39107/ using document root <none> and password file <none>
I20260927 03:58:04.950928 22968 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:04.950986 22968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:04.951085 22968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:04.952199 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "55e026c0789f4d4297128beb05f26061"
format_stamp: "Formatted at 2026-09-27 03:58:04 on dist-test-slave-96fg"
server_key: "0e86e72e382717629e8074ee49276b8e"
server_key_iv: "498e6d0bfea62eb574dd2ea1bca6d221"
server_key_version: "encryptionkey@0"
I20260927 03:58:04.952527 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "55e026c0789f4d4297128beb05f26061"
format_stamp: "Formatted at 2026-09-27 03:58:04 on dist-test-slave-96fg"
server_key: "0e86e72e382717629e8074ee49276b8e"
server_key_iv: "498e6d0bfea62eb574dd2ea1bca6d221"
server_key_version: "encryptionkey@0"
I20260927 03:58:04.954104 22968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:04.955132 22983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:04.955549 22968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:04.955665 22968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "55e026c0789f4d4297128beb05f26061"
format_stamp: "Formatted at 2026-09-27 03:58:04 on dist-test-slave-96fg"
server_key: "0e86e72e382717629e8074ee49276b8e"
server_key_iv: "498e6d0bfea62eb574dd2ea1bca6d221"
server_key_version: "encryptionkey@0"
I20260927 03:58:04.955783 22968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:04.959462 22968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:04.962191 22968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:04.962322 22968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:04.968961 22968 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:39061
I20260927 03:58:04.969097 23035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:39061 every 8 connection(s)
I20260927 03:58:04.969794 22968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:04.971167 23036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:04.973659 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 22968
I20260927 03:58:04.973784 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:04.974088 1225 external_mini_cluster.cc:1428] Setting key 24accd04120d3d48b4aa5ec4630d41a4
I20260927 03:58:04.974267 23036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061: Bootstrap starting.
I20260927 03:58:04.975132 23036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:04.975482 23036 log.cc:824] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:04.976522 23036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061: No bootstrap required, opened a new log
I20260927 03:58:04.977834 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:04.974723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34965 (local address 127.1.50.126:39061)
0927 03:58:04.974992 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:04.975004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:04.975192 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:58:04.975992 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:04.976004 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:04.976042 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:04.976208 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:04.977308 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:04.977312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:04.977312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:04.977317 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:04.977371 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:04.977384 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:04.977479 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:04.977546 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:58:04.977596 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260927 03:58:04.978897 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:04.979040 23036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } }
I20260927 03:58:04.979225 23036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:04.979259 23036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e026c0789f4d4297128beb05f26061, State: Initialized, Role: FOLLOWER
I20260927 03:58:04.979394 23036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } }
I20260927 03:58:04.979492 23036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:04.979524 23036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:04.979575 23036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:04.980352 23036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } }
I20260927 03:58:04.980499 23036 leader_election.cc:302] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e026c0789f4d4297128beb05f26061; no voters:
I20260927 03:58:04.980718 23036 leader_election.cc:288] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:04.980917 23047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:04.981051 23036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:04.981051 23047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [term 1 LEADER]: Becoming Leader. State: Replica: 55e026c0789f4d4297128beb05f26061, State: Running, Role: LEADER
I20260927 03:58:04.981357 23047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } }
I20260927 03:58:04.981842 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e026c0789f4d4297128beb05f26061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } } }
I20260927 03:58:04.981976 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:04.982250 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e026c0789f4d4297128beb05f26061. Latest consensus state: current_term: 1 leader_uuid: "55e026c0789f4d4297128beb05f26061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e026c0789f4d4297128beb05f26061" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39061 } } }
I20260927 03:58:04.982396 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:04.982751 23055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:04.982992 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:04.980812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55435 (local address 127.1.50.126:39061)
0927 03:58:04.980879 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:04.980882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:04.980991 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:58:04.981848 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:04.981849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:04.981855 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:04.981937 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:04.982667 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:04.982668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:04.982669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:04.982674 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:04.982691 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:04.982697 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:04.982740 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:04.982914 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:58:04.982929 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:58:04.983656 23055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:04.985914 23055 catalog_manager.cc:1347] Generated new cluster ID: 3b7dbde96e3945ab823d8c76ebfc10b9
I20260927 03:58:04.986044 23055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:05.006174 23055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:05.006426 23055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:05.014849 23055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e026c0789f4d4297128beb05f26061: Generated new TSK 0
I20260927 03:58:05.015156 23055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:05.020793 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 22968
2026-09-27T03:58:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:05 dist-test-slave-96fg krb5kdc[23067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:05 dist-test-slave-96fg krb5kdc[23067](info): set up 2 sockets
Sep 27 03:58:05 dist-test-slave-96fg krb5kdc[23067](info): commencing operation
krb5kdc: starting...
W20260927 03:58:05.981947 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:06.106640 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34179
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:39275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:06.206913 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:06.207448 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:06.207487 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:06.209786 23080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:06.209872 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:06.209892 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:06.209903 23080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:06.209915 23080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:06.211856 23080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:39275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:06.212127 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:06.212384 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:06.217749 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:06.217749 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:06.217752 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:06.218200 23080 server_base.cc:1034] running on GCE node
I20260927 03:58:06.218572 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:06.218931 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:06.220120 23080 hybrid_clock.cc:648] HybridClock initialized: now 1790481486220108 us; error 50 us; skew 500 ppm
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:06.223074 23080 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:06.223627 23080 webserver.cc:460] Webserver started at http://127.1.50.126:40967/ using document root <none> and password file <none>
I20260927 03:58:06.223848 23080 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:06.223898 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:06.224030 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:06.225662 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "e47d0349e80949ffa614fc378941c40a"
format_stamp: "Formatted at 2026-09-27 03:58:06 on dist-test-slave-96fg"
server_key: "643292e25be1fd262dbdba359e034cd8"
server_key_iv: "775e8d56a355ca6265af52567b035d30"
server_key_version: "encryptionkey@0"
I20260927 03:58:06.226665 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "e47d0349e80949ffa614fc378941c40a"
format_stamp: "Formatted at 2026-09-27 03:58:06 on dist-test-slave-96fg"
server_key: "643292e25be1fd262dbdba359e034cd8"
server_key_iv: "775e8d56a355ca6265af52567b035d30"
server_key_version: "encryptionkey@0"
I20260927 03:58:06.228816 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 03:58:06.229987 23095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:06.230367 23080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:06.230599 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "e47d0349e80949ffa614fc378941c40a"
format_stamp: "Formatted at 2026-09-27 03:58:06 on dist-test-slave-96fg"
server_key: "643292e25be1fd262dbdba359e034cd8"
server_key_iv: "775e8d56a355ca6265af52567b035d30"
server_key_version: "encryptionkey@0"
I20260927 03:58:06.230715 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:06.251905 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:06.254177 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:06.254329 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:06.259104 23080 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34179
I20260927 03:58:06.259152 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34179 every 8 connection(s)
I20260927 03:58:06.259545 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:06.260427 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:06.262444 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23080
I20260927 03:58:06.262562 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:06.262764 1225 external_mini_cluster.cc:1428] Setting key 4e18b8c871cbd70c0797901fb42966f2
I20260927 03:58:06.262992 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: Bootstrap starting.
I20260927 03:58:06.263600 23148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:06.263885 23148 log.cc:824] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:06.264457 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: No bootstrap required, opened a new log
I20260927 03:58:06.266255 23148 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } }
I20260927 03:58:06.266399 23148 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:06.266429 23148 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47d0349e80949ffa614fc378941c40a, State: Initialized, Role: FOLLOWER
I20260927 03:58:06.266556 23148 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } }
I20260927 03:58:06.266629 23148 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:06.266660 23148 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:06.266705 23148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:06.267243 23148 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } }
I20260927 03:58:06.267338 23148 leader_election.cc:302] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47d0349e80949ffa614fc378941c40a; no voters:
I20260927 03:58:06.267490 23148 leader_election.cc:288] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:06.267532 23153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:06.267737 23148 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:06.267863 23153 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [term 1 LEADER]: Becoming Leader. State: Replica: e47d0349e80949ffa614fc378941c40a, State: Running, Role: LEADER
I20260927 03:58:06.268182 23153 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } }
I20260927 03:58:06.269029 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e47d0349e80949ffa614fc378941c40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } } }
I20260927 03:58:06.269191 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:06.269384 23154 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47d0349e80949ffa614fc378941c40a. Latest consensus state: current_term: 1 leader_uuid: "e47d0349e80949ffa614fc378941c40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47d0349e80949ffa614fc378941c40a" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34179 } } }
I20260927 03:58:06.269587 23154 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:06.270217 23167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:06.270282 23167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:06.270337 23168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:06.270545 23168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:58:06.272832 23168 catalog_manager.cc:1347] Generated new cluster ID: c29e98b483e0448cbb7624e888a5ead7
I20260927 03:58:06.272887 23168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:06.275419 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:06.263443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43629 (local address 127.1.50.126:34179)
0927 03:58:06.263615 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:06.263622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:06.263651 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:58:06.264126 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:06.264133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:06.264180 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:06.264314 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:06.265027 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:06.265968 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:06.266842 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:06.267250 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:06.267346 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:06.271467 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:06.271473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:06.271473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:06.271489 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:06.273552 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:06.274048 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:06.274052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:06.274052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:06.274168 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:06.274384 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:06.274385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:06.274385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:06.274683 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:06.274821 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:06.274966 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:58:06.275101 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260927 03:58:06.276317 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:06.286419 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:06.278044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33141 (local address 127.1.50.126:34179)
0927 03:58:06.278081 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:06.278084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:06.278182 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:58:06.278254 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:06.278255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:06.278265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:06.278314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:06.278989 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:06.279880 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:06.280926 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:06.281254 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:06.281327 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:06.283506 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:06.283508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:06.283508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:06.283516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:06.285047 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:06.285454 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:06.285455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:06.285455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:06.285519 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:06.285796 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:06.285797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:06.285797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:06.285921 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:06.286014 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:06.286221 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:58:06.286304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:06.287762 23168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:06.289733 23168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:06.298717 23168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47d0349e80949ffa614fc378941c40a: Generated new TSK 0
I20260927 03:58:06.298938 23168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:06.304580 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23080
2026-09-27T03:58:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23179](info): set up 2 sockets
Sep 27 03:58:06 dist-test-slave-96fg krb5kdc[23179](info): commencing operation
krb5kdc: starting...
W20260927 03:58:07.274075 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:07 dist-test-slave-96fg krb5kdc[23179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:07.393684 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38051
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:38051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:07.472831 23192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:07.473037 23192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:07.473058 23192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:07.474200 23192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:07.474236 23192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:07.474247 23192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:07.474258 23192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:07.474269 23192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:07.475664 23192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:38051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:38051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:07.475875 23192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:07.476056 23192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:58:07.481045 23192 server_base.cc:1034] running on GCE node
W20260927 03:58:07.481094 23197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:07.481201 23198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:07.481038 23200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:07.481508 23192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:07.481824 23192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:07.482969 23192 hybrid_clock.cc:648] HybridClock initialized: now 1790481487482957 us; error 46 us; skew 500 ppm
Sep 27 03:58:07 dist-test-slave-96fg krb5kdc[23179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:07.485471 23192 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:07.485949 23192 webserver.cc:460] Webserver started at http://127.1.50.126:41099/ using document root <none> and password file <none>
I20260927 03:58:07.486189 23192 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:07.486239 23192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:07.486368 23192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:07.487440 23192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "901cc2988e18415a8a49fd8dde2ae64c"
format_stamp: "Formatted at 2026-09-27 03:58:07 on dist-test-slave-96fg"
server_key: "f4b7fbabbf16fd78a5915b8c4049b390"
server_key_iv: "d9737a99ee781cb5ed258b2b633c203f"
server_key_version: "encryptionkey@0"
I20260927 03:58:07.487756 23192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "901cc2988e18415a8a49fd8dde2ae64c"
format_stamp: "Formatted at 2026-09-27 03:58:07 on dist-test-slave-96fg"
server_key: "f4b7fbabbf16fd78a5915b8c4049b390"
server_key_iv: "d9737a99ee781cb5ed258b2b633c203f"
server_key_version: "encryptionkey@0"
I20260927 03:58:07.489027 23192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:07.489756 23207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:07.489988 23192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:07.490116 23192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "901cc2988e18415a8a49fd8dde2ae64c"
format_stamp: "Formatted at 2026-09-27 03:58:07 on dist-test-slave-96fg"
server_key: "f4b7fbabbf16fd78a5915b8c4049b390"
server_key_iv: "d9737a99ee781cb5ed258b2b633c203f"
server_key_version: "encryptionkey@0"
I20260927 03:58:07.490191 23192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:07.507890 23192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:07.510071 23192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:07.510216 23192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:07.514197 23192 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:38051
I20260927 03:58:07.514261 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:38051 every 8 connection(s)
I20260927 03:58:07.514573 23192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:07.515390 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:07.517690 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: Bootstrap starting.
I20260927 03:58:07.518294 23260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:07.518575 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23192
I20260927 03:58:07.518580 23260 log.cc:824] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:07.518702 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:07.518877 1225 external_mini_cluster.cc:1428] Setting key de9dd181953cd7528fbb71a66a6399ba
I20260927 03:58:07.519318 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: No bootstrap required, opened a new log
I20260927 03:58:07.521054 23260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } }
I20260927 03:58:07.521224 23260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:07.521260 23260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901cc2988e18415a8a49fd8dde2ae64c, State: Initialized, Role: FOLLOWER
I20260927 03:58:07.521430 23260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } }
I20260927 03:58:07.521517 23260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:07.521543 23260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:07.521588 23260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:07.522238 23260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } }
I20260927 03:58:07.522329 23260 leader_election.cc:302] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 901cc2988e18415a8a49fd8dde2ae64c; no voters:
I20260927 03:58:07.522486 23260 leader_election.cc:288] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:07.522534 23265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:07.522732 23265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [term 1 LEADER]: Becoming Leader. State: Replica: 901cc2988e18415a8a49fd8dde2ae64c, State: Running, Role: LEADER
I20260927 03:58:07.522737 23260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:07.522931 23265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } }
I20260927 03:58:07.524550 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "901cc2988e18415a8a49fd8dde2ae64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } } }
I20260927 03:58:07.524657 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:07.524670 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 901cc2988e18415a8a49fd8dde2ae64c. Latest consensus state: current_term: 1 leader_uuid: "901cc2988e18415a8a49fd8dde2ae64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901cc2988e18415a8a49fd8dde2ae64c" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 38051 } } }
I20260927 03:58:07.524714 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:07.524724 23280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:07.524770 23280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:07.524941 23282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:07.525151 23282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:07.526366 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:07.519396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40071 (local address 127.1.50.126:38051)
0927 03:58:07.519561 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:07.519569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:07.519595 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:58:07.521028 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:07.521044 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:07.521108 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:07.521319 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:07.521500 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:07.523082 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:07.524278 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:07.524873 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:07.524985 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:07.525629 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:07.525633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:07.525634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:07.525851 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:07.525890 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:07.525906 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:07.525978 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:07.526090 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:58:07.526157 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260927 03:58:07.527175 23282 catalog_manager.cc:1347] Generated new cluster ID: e35ac801c935467f9168b295b3cf0b65
I20260927 03:58:07.527218 23282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:07.527323 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:07.532531 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:07.528810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56599 (local address 127.1.50.126:38051)
0927 03:58:07.528847 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:07.528848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:07.528960 (+ 112us) server_negotiation.cc:407] Connection header received
0927 03:58:07.529050 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:07.529051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:07.529061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:07.529088 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:07.529271 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:07.530134 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:07.531172 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:07.531460 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:07.531503 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:07.532273 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:07.532274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:07.532275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:07.532280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:07.532294 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:07.532299 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:07.532324 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:07.532423 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:58:07.532449 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:07.537653 23282 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:07.538224 23282 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:07.548509 23282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 901cc2988e18415a8a49fd8dde2ae64c: Generated new TSK 0
I20260927 03:58:07.548717 23282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:07.552687 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23192
2026-09-27T03:58:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:07 dist-test-slave-96fg krb5kdc[23291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:07 dist-test-slave-96fg krb5kdc[23291](info): set up 2 sockets
Sep 27 03:58:07 dist-test-slave-96fg krb5kdc[23291](info): commencing operation
krb5kdc: starting...
W20260927 03:58:08.517889 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:08 dist-test-slave-96fg krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:08.638310 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34015
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:34015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:08.718714 23304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:08.718933 23304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:08.718957 23304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:08.720114 23304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:08.720155 23304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:08.720167 23304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:08.720178 23304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:08.720189 23304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:08.721637 23304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:34015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:34015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:08.721843 23304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:08.722095 23304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:08.727028 23312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:08.727036 23309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:08.727037 23310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:08.727197 23304 server_base.cc:1034] running on GCE node
I20260927 03:58:08.727452 23304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:08.727706 23304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:08.728849 23304 hybrid_clock.cc:648] HybridClock initialized: now 1790481488728818 us; error 46 us; skew 500 ppm
Sep 27 03:58:08 dist-test-slave-96fg krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:08.734588 23304 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:08.735181 23304 webserver.cc:460] Webserver started at http://127.1.50.126:45203/ using document root <none> and password file <none>
I20260927 03:58:08.735430 23304 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:08.735482 23304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:08.735615 23304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:08.736712 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "bd4e6ffadefb45b581ab6a818efe58ae"
format_stamp: "Formatted at 2026-09-27 03:58:08 on dist-test-slave-96fg"
server_key: "e48008e744c28922f34ff42651f09e79"
server_key_iv: "3aaeabc97b22146ae5e6fa102cec3990"
server_key_version: "encryptionkey@0"
I20260927 03:58:08.737054 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "bd4e6ffadefb45b581ab6a818efe58ae"
format_stamp: "Formatted at 2026-09-27 03:58:08 on dist-test-slave-96fg"
server_key: "e48008e744c28922f34ff42651f09e79"
server_key_iv: "3aaeabc97b22146ae5e6fa102cec3990"
server_key_version: "encryptionkey@0"
I20260927 03:58:08.738831 23304 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 03:58:08.740037 23319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:08.740288 23304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:08.740401 23304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "bd4e6ffadefb45b581ab6a818efe58ae"
format_stamp: "Formatted at 2026-09-27 03:58:08 on dist-test-slave-96fg"
server_key: "e48008e744c28922f34ff42651f09e79"
server_key_iv: "3aaeabc97b22146ae5e6fa102cec3990"
server_key_version: "encryptionkey@0"
I20260927 03:58:08.740490 23304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:08.744334 23304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:08.746599 23304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:08.746735 23304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:08.750478 23304 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:34015
I20260927 03:58:08.750576 23371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:34015 every 8 connection(s)
I20260927 03:58:08.750847 23304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:08.751813 23372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:08.752873 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23304
I20260927 03:58:08.752982 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:08.753152 1225 external_mini_cluster.cc:1428] Setting key ceaa22cd6ee8a308d965de0c7bdab453
I20260927 03:58:08.754506 23372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae: Bootstrap starting.
I20260927 03:58:08.755133 23372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:08.755436 23372 log.cc:824] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:08.755589 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:08.753629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43387 (local address 127.1.50.126:34015)
0927 03:58:08.753810 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:08.753820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:08.753853 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:58:08.754368 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:08.754376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:08.754408 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:08.754552 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:08.755131 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:08.755135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:08.755135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:08.755140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:08.755182 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:08.755197 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:08.755302 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:08.755336 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:58:08.755377 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260927 03:58:08.756139 23372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae: No bootstrap required, opened a new log
I20260927 03:58:08.756485 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:08.758147 23372 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } }
I20260927 03:58:08.758323 23372 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:08.758363 23372 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4e6ffadefb45b581ab6a818efe58ae, State: Initialized, Role: FOLLOWER
I20260927 03:58:08.758502 23372 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } }
I20260927 03:58:08.758598 23372 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:08.758680 23372 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:08.758736 23372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:08.758786 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:08.757729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50265 (local address 127.1.50.126:34015)
0927 03:58:08.757768 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:08.757770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:08.757822 (+ 52us) server_negotiation.cc:407] Connection header received
0927 03:58:08.757873 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:08.757874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:08.757878 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:08.757938 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:08.758559 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:08.758560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:08.758561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:08.758565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:08.758580 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:08.758584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:08.758642 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:08.758688 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 03:58:08.758700 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:08.759366 23372 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } }
I20260927 03:58:08.759455 23372 leader_election.cc:302] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4e6ffadefb45b581ab6a818efe58ae; no voters:
I20260927 03:58:08.759596 23372 leader_election.cc:288] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:08.759673 23382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:08.759825 23372 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:08.759840 23382 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [term 1 LEADER]: Becoming Leader. State: Replica: bd4e6ffadefb45b581ab6a818efe58ae, State: Running, Role: LEADER
I20260927 03:58:08.759982 23382 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } }
I20260927 03:58:08.760406 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4e6ffadefb45b581ab6a818efe58ae. Latest consensus state: current_term: 1 leader_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } } }
I20260927 03:58:08.760494 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:08.760412 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4e6ffadefb45b581ab6a818efe58ae" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 34015 } } }
I20260927 03:58:08.760787 23391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:08.760876 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:08.761305 23391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:08.762871 23391 catalog_manager.cc:1347] Generated new cluster ID: a01b55ea29e34b67b82fdaba52f616dd
I20260927 03:58:08.762921 23391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:08.780997 23391 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:08.781186 23391 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:08.788483 23391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd4e6ffadefb45b581ab6a818efe58ae: Generated new TSK 0
I20260927 03:58:08.788702 23391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:08.795970 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23304
2026-09-27T03:58:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1243 ms)
[----------] 18 tests from AuthTokenIssuingTest (22911 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:08 dist-test-slave-96fg krb5kdc[23403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:08 dist-test-slave-96fg krb5kdc[23403](info): set up 2 sockets
Sep 27 03:58:08 dist-test-slave-96fg krb5kdc[23403](info): commencing operation
krb5kdc: starting...
W20260927 03:58:09.754001 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:09 dist-test-slave-96fg krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:09.877460 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40513
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:09.960930 23416 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:09.961192 23416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:09.961220 23416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:09.961320 23416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:09.962709 23416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:09.962776 23416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:09.962797 23416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:09.962813 23416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:09.962827 23416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:09.964581 23416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:09.965209 23416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:09.965526 23416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:09.970773 23424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:09.970806 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:09.970773 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:09.970875 23416 server_base.cc:1034] running on GCE node
I20260927 03:58:09.971314 23416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:09.971619 23416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:09.972774 23416 hybrid_clock.cc:648] HybridClock initialized: now 1790481489972728 us; error 66 us; skew 500 ppm
Sep 27 03:58:09 dist-test-slave-96fg krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:09.975373 23416 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:09.975859 23416 webserver.cc:460] Webserver started at http://127.1.50.126:32845/ using document root <none> and password file <none>
I20260927 03:58:09.976090 23416 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:09.976143 23416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:09.976275 23416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:09.977419 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "47583cb918ad4176a43cbcafd7231150"
format_stamp: "Formatted at 2026-09-27 03:58:09 on dist-test-slave-96fg"
server_key: "54758fba5d2ad5cefeef59f5305b0019"
server_key_iv: "98768dacbcbca2cb1212633ed344ba98"
server_key_version: "encryptionkey@0"
I20260927 03:58:09.977748 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "47583cb918ad4176a43cbcafd7231150"
format_stamp: "Formatted at 2026-09-27 03:58:09 on dist-test-slave-96fg"
server_key: "54758fba5d2ad5cefeef59f5305b0019"
server_key_iv: "98768dacbcbca2cb1212633ed344ba98"
server_key_version: "encryptionkey@0"
I20260927 03:58:09.979187 23416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:58:09.980054 23431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:09.980307 23416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:09.980407 23416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "47583cb918ad4176a43cbcafd7231150"
format_stamp: "Formatted at 2026-09-27 03:58:09 on dist-test-slave-96fg"
server_key: "54758fba5d2ad5cefeef59f5305b0019"
server_key_iv: "98768dacbcbca2cb1212633ed344ba98"
server_key_version: "encryptionkey@0"
I20260927 03:58:09.980508 23416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:09.990801 23416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:09.992985 23416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:09.993137 23416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:09.997586 23416 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:40513
I20260927 03:58:09.997650 23483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:40513 every 8 connection(s)
I20260927 03:58:09.998136 23416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:09.999318 23484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:10.001448 23484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.002576 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23416
I20260927 03:58:10.002691 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:10.002871 1225 external_mini_cluster.cc:1428] Setting key 7e5fa5907700ffe4d4c573df1a712a33
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.125@KRBTEST.COM; defaulting to no policy
I20260927 03:58:10.012466 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.002357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.1.50.126:40513)
0927 03:58:10.003435 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.003440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.003457 (+ 17us) server_negotiation.cc:407] Connection header received
0927 03:58:10.003534 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.003537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.003557 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.003637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.004221 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.005068 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.006162 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.006573 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.006687 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.009099 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.009107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.009107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.009113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.010733 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.011163 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.011166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.011166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.011266 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.011550 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.011551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.011552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.011792 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.011879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.012093 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.012257 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":1041,"thread_start_us":115,"threads_started":1}
I20260927 03:58:10.012487 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.002357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40513 (local address 127.0.0.1:47796)
0927 03:58:10.002569 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.002619 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.003250 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.003669 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.003671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.003957 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.004118 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.004122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.005160 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.005161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.006077 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.006081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.006707 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.006709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.006796 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.007312 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.007323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.008983 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.010829 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.010836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.010839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.011066 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.011347 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.011348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.011348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.011463 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.011901 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.011908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.011971 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.012055 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.012206 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":4,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.125@KRBTEST.COM" created.
I20260927 03:58:10.013204 23484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.013518 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.013414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35176)
0927 03:58:10.013456 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.013498 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:58:10.013650 23432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
I20260927 03:58:10.014110 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.014006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35178)
0927 03:58:10.014062 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.014089 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 03:58:10.014201 23484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:46301: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
I20260927 03:58:10.029578 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:46301
--webserver_interface=127.1.50.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:10.063427 23484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } attempt: 1
I20260927 03:58:10.063974 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.063801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35192)
0927 03:58:10.063892 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.063941 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 03:58:10.064322 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.064213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35206)
0927 03:58:10.064271 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.064301 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:58:10.064425 23484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:46301: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
W20260927 03:58:10.113955 23491 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:10.114248 23491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:10.114274 23491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:10.114302 23491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:10.115530 23491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:10.115578 23491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:10.115597 23491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:10.115612 23491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:10.115629 23491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:10.117238 23491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.50.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:10.117501 23491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:10.117733 23491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:10.122756 23497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:10.122829 23496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:10.122826 23491 server_base.cc:1034] running on GCE node
W20260927 03:58:10.122756 23499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:10.123282 23491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:10.123620 23491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:10.124799 23491 hybrid_clock.cc:648] HybridClock initialized: now 1790481490124779 us; error 37 us; skew 500 ppm
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:10.127588 23491 init.cc:377] Logged in from keytab as kudu/127.1.50.125@KRBTEST.COM (short username kudu)
I20260927 03:58:10.128090 23491 webserver.cc:460] Webserver started at http://127.1.50.125:42845/ using document root <none> and password file <none>
I20260927 03:58:10.128352 23491 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:10.128409 23491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:10.128533 23491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:10.129791 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data/instance:
uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "683ebcad5b7730737339dcec4e0bc9c6"
server_key_iv: "77214fbc8c159f3d2c41cfd78c379570"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.130153 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal/instance:
uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "683ebcad5b7730737339dcec4e0bc9c6"
server_key_iv: "77214fbc8c159f3d2c41cfd78c379570"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.131459 23491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:10.132244 23506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:10.132477 23491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:10.132587 23491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "683ebcad5b7730737339dcec4e0bc9c6"
server_key_iv: "77214fbc8c159f3d2c41cfd78c379570"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.132607 23484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } attempt: 2
I20260927 03:58:10.132653 23491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:10.133051 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.132886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35210)
0927 03:58:10.132952 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.133013 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:58:10.133378 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.133277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35214)
0927 03:58:10.133307 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.133354 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 03:58:10.133481 23484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:46301: Network error: Client connection negotiation failed: client connection to 127.1.50.125:46301: connect: Connection refused (error 111)
I20260927 03:58:10.146373 23491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:10.148564 23491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:10.148682 23491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:10.152541 23491 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.125:46301
I20260927 03:58:10.152585 23558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.125:46301 every 8 connection(s)
I20260927 03:58:10.152935 23491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
I20260927 03:58:10.153960 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:10.154675 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23491
I20260927 03:58:10.154778 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-1/wal/instance
I20260927 03:58:10.154951 1225 external_mini_cluster.cc:1428] Setting key 42149687715d1a595913f6c66421e3ec
I20260927 03:58:10.155968 23559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.50.124@KRBTEST.COM; defaulting to no policy
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.124@KRBTEST.COM" created.
I20260927 03:58:10.166343 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.156807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.1.50.126:40513)
0927 03:58:10.156882 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.156885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.157097 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:58:10.157658 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.157660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.157670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.157718 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.158604 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.159081 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.160568 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.160773 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.160840 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.163491 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.163493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.163493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.163502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.164847 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.165248 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.165249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.165250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.165305 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.165560 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.165561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.165562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.165721 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.165797 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.166176 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.166270 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:58:10.166613 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.156806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40513 (local address 127.0.0.1:47800)
0927 03:58:10.156981 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.157018 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.157548 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.157796 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.157801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.158231 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.158504 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.158510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.159174 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.159176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.160453 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.160457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.160891 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.160893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.161054 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.161675 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.161686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.163378 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.164926 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.164934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.164936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.165156 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.165370 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.165370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.165371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.165487 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.165796 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.165810 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.165985 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.166134 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.166312 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260927 03:58:10.167218 23559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:10.174036 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.167540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35220)
0927 03:58:10.167587 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.167598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.167636 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.167855 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.167856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.167989 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.168051 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.168053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.168875 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.168876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.169374 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.169375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.169814 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.169815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.169874 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.170304 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.170309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.171432 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.173052 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.173053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.173054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.173206 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.173381 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.173382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.173382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.173434 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.173690 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.173690 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.173734 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.173795 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.173901 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:58:10.174094 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.167581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.1.50.125:46301)
0927 03:58:10.167722 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.167726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.167750 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:58:10.167772 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.167773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.167784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.167846 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.168113 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.168802 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.169439 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.169753 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.169811 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.171508 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.171517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.171517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.171524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.172970 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.173275 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.173278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.173278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.173329 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.173483 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.173484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.173484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.173631 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.173701 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.173831 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.173953 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260927 03:58:10.174911 23559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.175305 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.175192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51098)
0927 03:58:10.175225 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.175280 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 03:58:10.175467 23510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
I20260927 03:58:10.175863 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.175754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51108)
0927 03:58:10.175816 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.175842 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 03:58:10.175999 23559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42183: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
I20260927 03:58:10.182372 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:42183
--webserver_interface=127.1.50.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:10.224678 23484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } attempt: 3
I20260927 03:58:10.225240 23559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } attempt: 1
I20260927 03:58:10.225630 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.225502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51112)
0927 03:58:10.225551 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.225592 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:58:10.226387 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.226250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51114)
0927 03:58:10.226339 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.226365 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 03:58:10.226467 23559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42183: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:10.232016 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.225090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35236)
0927 03:58:10.225164 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.225174 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.225222 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.225358 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.225358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.225517 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.225604 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.225606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.226587 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.226588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.227104 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.227106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.227593 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.227594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.227663 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.228078 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.228082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.229240 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.231002 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.231004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.231004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.231185 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.231408 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.231409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.231409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.231476 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.231755 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.231756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.231798 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.231859 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.231946 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 03:58:10.232064 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.225167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.1.50.125:46301)
0927 03:58:10.225230 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.225232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.225257 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:58:10.225283 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.225284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.225297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.225351 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.225687 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.226501 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.227175 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.227512 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.227578 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.229349 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.229350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.229351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.229358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.230900 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.231254 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.231255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.231255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.231314 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.231539 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.231539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.231540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.231671 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.231731 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.231894 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.231976 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:10.232456 23484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.232790 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.232691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51126)
0927 03:58:10.232738 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.232767 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:58:10.233130 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.233051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51142)
0927 03:58:10.233092 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.233115 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:58:10.233203 23484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42183: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
I20260927 03:58:10.264343 23484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } attempt: 1
I20260927 03:58:10.264815 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.264696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51156)
0927 03:58:10.264746 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.264785 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:58:10.265154 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.265069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51172)
0927 03:58:10.265109 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.265136 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:58:10.265240 23484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42183: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
W20260927 03:58:10.265296 23566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:10.265535 23566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:10.265570 23566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:10.265609 23566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:10.266886 23566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:10.266938 23566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:10.266957 23566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:10.266973 23566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:10.266989 23566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:10.268616 23566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:40513,127.1.50.125:46301,127.1.50.124:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.50.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:10.268886 23566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:10.269099 23566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:10.274237 23571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:10.274310 23574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:10.274289 23566 server_base.cc:1034] running on GCE node
W20260927 03:58:10.274260 23572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:10.274765 23566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:10.275053 23566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:10.276198 23566 hybrid_clock.cc:648] HybridClock initialized: now 1790481490276186 us; error 33 us; skew 500 ppm
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:10.278609 23566 init.cc:377] Logged in from keytab as kudu/127.1.50.124@KRBTEST.COM (short username kudu)
I20260927 03:58:10.279063 23566 webserver.cc:460] Webserver started at http://127.1.50.124:38993/ using document root <none> and password file <none>
I20260927 03:58:10.279268 23566 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:10.279314 23566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:10.279433 23566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:10.280546 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data/instance:
uuid: "78a83133791f4badbbb1514ec4841542"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "7f70b9c2646ee38c63a71d67332f4081"
server_key_iv: "addc9b0fd420ec44fdc6b89f53c4bf3e"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.280875 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal/instance:
uuid: "78a83133791f4badbbb1514ec4841542"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "7f70b9c2646ee38c63a71d67332f4081"
server_key_iv: "addc9b0fd420ec44fdc6b89f53c4bf3e"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.282174 23566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:10.282971 23581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:10.283176 23566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:10.283264 23566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "78a83133791f4badbbb1514ec4841542"
format_stamp: "Formatted at 2026-09-27 03:58:10 on dist-test-slave-96fg"
server_key: "7f70b9c2646ee38c63a71d67332f4081"
server_key_iv: "addc9b0fd420ec44fdc6b89f53c4bf3e"
server_key_version: "encryptionkey@0"
I20260927 03:58:10.283346 23566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:10.294602 23559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } attempt: 2
I20260927 03:58:10.295006 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.294900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51174)
0927 03:58:10.294945 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.294976 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:58:10.295449 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.295341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51178)
0927 03:58:10.295399 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.295431 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:58:10.295552 23559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:42183: Network error: Client connection negotiation failed: client connection to 127.1.50.124:42183: connect: Connection refused (error 111)
I20260927 03:58:10.296128 23566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:10.298286 23566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:10.298415 23566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:10.302222 23566 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.124:42183
I20260927 03:58:10.302253 23633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.124:42183 every 8 connection(s)
I20260927 03:58:10.302583 23566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
I20260927 03:58:10.303541 23634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:10.305089 23634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.307360 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23566
I20260927 03:58:10.307464 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481222547165-1225-0/minicluster-data/master-2/wal/instance
I20260927 03:58:10.307662 1225 external_mini_cluster.cc:1428] Setting key 555a93e84e44c9a6498d374d19056aab
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:10.316401 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.305770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.1.50.126:40513)
0927 03:58:10.305857 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.305859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.306094 (+ 235us) server_negotiation.cc:407] Connection header received
0927 03:58:10.306573 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.306574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.306585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.306665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.307266 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.307938 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.309326 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.309520 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.309578 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.312914 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.312916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.312916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.312924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.314509 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.315124 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.315128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.315128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.315221 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.315620 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.315622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.315623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.315839 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.315919 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.316239 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.316327 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 03:58:10.316656 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.305743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:40513 (local address 127.0.0.1:47802)
0927 03:58:10.305927 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.305964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.306471 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.306718 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.306721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.306972 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.307164 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.307167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.308049 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.308050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.309251 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.309254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.309606 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.309607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.309733 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.310269 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.310280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.312780 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.314608 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.314616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.314618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.314975 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.315335 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.315336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.315337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.315502 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.315941 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.315948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.316087 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.316185 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.316335 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260927 03:58:10.317219 23634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.317939 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.308041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.1.50.124:42183)
0927 03:58:10.308208 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.308211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.308243 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:58:10.308308 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.308309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.308325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.308392 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.309123 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.309823 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.310791 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.311114 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.311181 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.314245 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.314255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.314256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.314264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.316217 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.316899 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.316901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.316902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.316964 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.317147 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.317148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.317148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.317303 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.317443 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.317699 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.317851 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:58:10.324020 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.317467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:46301 (local address 127.0.0.1:35248)
0927 03:58:10.317516 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.317527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.317565 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.317709 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.317710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.317884 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.317942 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.317944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.318799 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.318800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.319531 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.319533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.319866 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.319867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.319934 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.320322 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.320326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.321647 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.323063 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.323063 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.323064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.323316 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.323499 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.323499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.323499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.323550 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.323788 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.323789 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.323833 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.323887 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.323957 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:58:10.324088 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.317546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.1.50.125:46301)
0927 03:58:10.317606 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.317608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.317616 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:10.317639 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.317640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.317651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.317700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.318008 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.318630 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.319597 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.319809 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.319871 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.321721 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.321723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.321723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.321730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.322993 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.323379 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.323380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.323380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.323437 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.323599 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.323600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.323600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.323726 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.323790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.323927 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.323998 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:58:10.324409 23634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:10.325428 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.318695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.1.50.126:40513)
0927 03:58:10.318729 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.318731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.318738 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:10.318795 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.318796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.318802 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.318837 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.319424 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.319866 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.320636 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.320880 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.320945 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.322957 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.322958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.322959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.322965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.324438 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.324779 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.324780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.324780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.324836 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.324994 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.324994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.324995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.325096 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.325239 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.325276 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.325345 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:10.331001 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.324643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51194)
0927 03:58:10.324685 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.324695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.324728 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.324817 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.324817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.324933 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.324958 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.324959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.325557 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.325558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.326249 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.326251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.326604 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.326605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.326643 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.326936 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.326940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.328069 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.329756 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.329757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.329758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.329976 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.330213 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.330214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.330214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.330284 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.330592 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.330593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.330641 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.330734 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.330853 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260927 03:58:10.331049 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.324708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.1.50.124:42183)
0927 03:58:10.324742 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.324744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.324750 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:58:10.324766 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.324766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.324776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.324816 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.325013 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.325497 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.326308 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.326516 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.326565 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.328159 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.328161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.328161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.328167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.329639 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.330100 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.330100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.330101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.330147 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.330369 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.330370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.330371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.330515 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.330578 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.330753 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.330900 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":20}
I20260927 03:58:10.332667 23634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: Bootstrap starting.
I20260927 03:58:10.333669 23634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:10.333916 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.326061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.1.50.125:46301)
0927 03:58:10.326092 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.326094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.326102 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:10.326132 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.326133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.326141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.326187 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.326571 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.327224 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.327926 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.328208 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.328259 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.330922 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.330923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.330923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.330930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.332620 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.333008 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.333010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.333010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.333076 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.333309 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.333310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.333310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.333442 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.333604 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.333671 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.333802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:58:10.334129 23634 log.cc:824] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:10.335047 23634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: No bootstrap required, opened a new log
I20260927 03:58:10.335759 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:10.336856 23634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.337003 23634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a83133791f4badbbb1514ec4841542, State: Initialized, Role: FOLLOWER
I20260927 03:58:10.337090 23634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.337440 23634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:10.337389 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.337502 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.337265 23458 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "47583cb918ad4176a43cbcafd7231150"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47806
W20260927 03:58:10.339380 23651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:10.339445 23651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:10.340418 23484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } attempt: 2
I20260927 03:58:10.339658 23566 master_runner.cc:186] Error getting master registration for 127.1.50.126:40513: OK, instance_id { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" instance_seqno: 1790481489996200 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:58:10.342572 23566 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:10.348737 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.340734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51210)
0927 03:58:10.340814 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.340827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.340886 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.341077 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.341078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.341244 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.341286 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.341287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.342242 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.342243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.343018 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.343021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.343606 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.343607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.343665 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.344146 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.344151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.345730 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.347602 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.347603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.347604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.347793 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.348004 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.348005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.348005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.348065 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.348389 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.348390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.348442 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.348511 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.348625 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 03:58:10.348737 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.340765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.1.50.124:42183)
0927 03:58:10.340827 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.340829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.340884 (+ 55us) server_negotiation.cc:407] Connection header received
0927 03:58:10.340959 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.340960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.340971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.341061 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.341357 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.342154 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.343184 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.343501 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.343590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.345829 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.345830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.345831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.345839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.347515 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.347866 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.347867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.347868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.347931 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.348120 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.348121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.348122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.348315 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.348392 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.348539 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.348626 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:10.350548 23484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: Bootstrap starting.
I20260927 03:58:10.351289 23484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:10.351616 23484 log.cc:824] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:10.352512 23484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: No bootstrap required, opened a new log
I20260927 03:58:10.353910 23484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.354085 23484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47583cb918ad4176a43cbcafd7231150, State: Initialized, Role: FOLLOWER
I20260927 03:58:10.354172 23484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.354502 23484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:10.354470 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.354559 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.354629 23458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:58:10.354687 23458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:10.355404 23458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.355739 23458 leader_election.cc:288] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [CANDIDATE]: Term 1 election: Requested vote from peers f987e0c6e8374ea4a5ffdb6b8e006e35 (127.1.50.125:46301), 78a83133791f4badbbb1514ec4841542 (127.1.50.124:42183)
I20260927 03:58:10.356086 23608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47583cb918ad4176a43cbcafd7231150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78a83133791f4badbbb1514ec4841542"
I20260927 03:58:10.356246 23608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:10.357123 23608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47583cb918ad4176a43cbcafd7231150 in term 1.
I20260927 03:58:10.357414 23435 leader_election.cc:302] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47583cb918ad4176a43cbcafd7231150, 78a83133791f4badbbb1514ec4841542; no voters:
I20260927 03:58:10.357715 23654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:10.357853 23654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [term 1 LEADER]: Becoming Leader. State: Replica: 47583cb918ad4176a43cbcafd7231150, State: Running, Role: LEADER
I20260927 03:58:10.357940 23654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.358397 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47583cb918ad4176a43cbcafd7231150. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.358469 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:10.359383 23416 master_runner.cc:186] Error getting master registration for 127.1.50.125:46301: OK, instance_id { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" instance_seqno: 1790481490151405 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:58:10.362718 23416 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:58:10.363284 23665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:10.363343 23665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:10.367067 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.357453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.1.50.126:40513)
0927 03:58:10.357507 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.357509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.357516 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:10.357534 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.357535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.357543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.357585 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.358955 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.359676 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.361457 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.361714 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.361750 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.363692 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.363694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.363694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.363701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.365830 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.366248 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.366249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.366249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.366302 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.366500 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.366501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.366502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.366628 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.366714 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.366897 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.366975 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:58:10.367139 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.357325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.1.50.125:46301)
0927 03:58:10.357402 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.357403 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.357905 (+ 502us) server_negotiation.cc:407] Connection header received
0927 03:58:10.358184 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.358185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.358198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.358240 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.359267 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.359922 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.361233 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.361546 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.361583 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.364009 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.364010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.364010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.364029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.365510 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.366052 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.366054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.366054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.366115 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.366431 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.366432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.366433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.366572 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.366643 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.366955 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.367031 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:58:10.367729 23675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:10.368046 23675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:10.368485 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.357024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.1.50.124:42183)
0927 03:58:10.357065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.357067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.357417 (+ 350us) server_negotiation.cc:407] Connection header received
0927 03:58:10.357504 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.357506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.357517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.357545 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.360672 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.361375 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.363769 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.364062 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.364101 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.365642 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.365644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.365644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.365652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.367228 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.367618 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.367619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.367620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.367679 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.367887 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.367888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.367888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.368012 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.368085 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.368302 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.368391 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:10.370218 23608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [term 1 FOLLOWER]: Refusing update from remote peer 47583cb918ad4176a43cbcafd7231150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:10.370512 23654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:10.371794 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47583cb918ad4176a43cbcafd7231150. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.371889 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.372893 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.372983 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:10.373304 23677 mvcc.cc:205] Tried to move back new op lower bound from 7333812184553783296 to 7333812184506863616. Current Snapshot: MvccSnapshot[applied={T|T < 7333812184553783296}]
I20260927 03:58:10.373646 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.373736 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.374058 23675 catalog_manager.cc:1347] Generated new cluster ID: 0a7c29ff42df40758a848548aea85703
I20260927 03:58:10.374101 23675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:10.386664 23559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } attempt: 3
Sep 27 03:58:10 dist-test-slave-96fg krb5kdc[23403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:10.393230 23675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:10.393982 23675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:10.394881 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.387124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:42183 (local address 127.0.0.1:51222)
0927 03:58:10.387161 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:10.387172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:10.387212 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:10.387444 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:10.387445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:10.387634 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:10.387684 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.387686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.388554 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.388555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.389329 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:10.389331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.389817 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.389818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:10.389890 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.390392 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:10.390398 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:10.391882 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:10.393721 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.393722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.393723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.393946 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.394206 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:10.394206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:10.394207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:10.394289 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:10.394567 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:10.394568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:10.394624 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 03:58:10.394674 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:58:10.394784 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 03:58:10.395987 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:10.387279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.1.50.124:42183)
0927 03:58:10.387320 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.387322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.387329 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:10.387348 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.387349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.387359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.387399 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.387767 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.388473 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.389416 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:10.389686 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:10.389740 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:10.391987 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:10.391989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:10.391989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:10.391997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:10.393603 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.394063 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.394064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.394065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.394122 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:10.394355 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:10.394356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:10.394356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:10.394487 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:10.394533 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:10.395818 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0927 03:58:10.395897 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:10.397379 23559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: Bootstrap starting.
I20260927 03:58:10.398120 23559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:10.398404 23559 log.cc:824] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:10.399416 23559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: No bootstrap required, opened a new log
I20260927 03:58:10.400825 23559 raft_consensus.cc:348] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.400944 23559 raft_consensus.cc:729] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f987e0c6e8374ea4a5ffdb6b8e006e35, State: Initialized, Role: FOLLOWER
I20260927 03:58:10.401005 23559 consensus_queue.cc:260] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } }
I20260927 03:58:10.401268 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.401345 23559 sys_catalog.cc:564] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:10.401360 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.402467 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47583cb918ad4176a43cbcafd7231150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35"
I20260927 03:58:10.403105 23675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150: Generated new TSK 0
I20260927 03:58:10.402612 23532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:10.403292 23675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:10.404336 23532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [term 1 FOLLOWER]: Refusing update from remote peer 47583cb918ad4176a43cbcafd7231150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:10.404731 23673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:10.406253 23533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47583cb918ad4176a43cbcafd7231150 in term 1.
W20260927 03:58:10.406801 23694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:10.406858 23694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:10.407797 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.407886 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:10.410787 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47583cb918ad4176a43cbcafd7231150. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.410887 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:58:10.432458 23638 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.431423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.1.50.124:42183)
0927 03:58:10.431469 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.431471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.431799 (+ 328us) server_negotiation.cc:407] Connection header received
0927 03:58:10.431892 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.431893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.431906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.431935 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.432390 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51226: BlockingRecv error: recv got EOF from 127.0.0.1:51226 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:58:10.433665 23487 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.432818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.1.50.126:40513)
0927 03:58:10.432855 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.432857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.433149 (+ 292us) server_negotiation.cc:407] Connection header received
0927 03:58:10.433238 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.433239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.433250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.433278 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.433600 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47828: BlockingRecv error: recv got EOF from 127.0.0.1:47828 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:58:10.434435 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.433033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.1.50.125:46301)
0927 03:58:10.433070 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.433071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.433921 (+ 850us) server_negotiation.cc:407] Connection header received
0927 03:58:10.434008 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.434009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.434039 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.434068 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.434373 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35268: BlockingRecv error: recv got EOF from 127.0.0.1:35268 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:58:10.436625 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.435904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.1.50.125:46301)
0927 03:58:10.435936 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.435938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.436108 (+ 170us) server_negotiation.cc:407] Connection header received
0927 03:58:10.436201 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.436202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.436219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.436245 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.436558 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:58:10.440014 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.439127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.1.50.125:46301)
0927 03:58:10.439164 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.439166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.439462 (+ 296us) server_negotiation.cc:407] Connection header received
0927 03:58:10.439563 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.439564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.439575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.439605 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.439962 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35284: BlockingRecv error: recv got EOF from 127.0.0.1:35284 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:58:10.443989 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.443215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.1.50.125:46301)
0927 03:58:10.443250 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.443252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.443474 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:58:10.443565 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.443566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.443577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.443604 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.443935 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35290: BlockingRecv error: recv got EOF from 127.0.0.1:35290 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:58:10.450006 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.449191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.1.50.125:46301)
0927 03:58:10.449228 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.449230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.449469 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:58:10.449566 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.449567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.449578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.449606 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.449952 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35302: BlockingRecv error: recv got EOF from 127.0.0.1:35302 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:58:10.460422 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.459465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.1.50.125:46301)
0927 03:58:10.459513 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.459515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.459793 (+ 278us) server_negotiation.cc:407] Connection header received
0927 03:58:10.459906 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.459907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.459920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.459950 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.460359 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35314: BlockingRecv error: recv got EOF from 127.0.0.1:35314 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:58:10.478837 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.477875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.1.50.125:46301)
0927 03:58:10.477918 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.477921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.478223 (+ 302us) server_negotiation.cc:407] Connection header received
0927 03:58:10.478326 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.478328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.478341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.478371 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.478774 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35316: BlockingRecv error: recv got EOF from 127.0.0.1:35316 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:58:10.513113 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.512271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.1.50.125:46301)
0927 03:58:10.512326 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.512328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.512563 (+ 235us) server_negotiation.cc:407] Connection header received
0927 03:58:10.512653 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.512654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.512667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.512694 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.513052 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 03:58:10.579156 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.578504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.1.50.125:46301)
0927 03:58:10.578557 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.578559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.578616 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:58:10.578693 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.578694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.578706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.578761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.579085 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35334: BlockingRecv error: recv got EOF from 127.0.0.1:35334 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 03:58:10.709283 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.708480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.1.50.125:46301)
0927 03:58:10.708540 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.708542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.708554 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:10.708582 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.708583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.708594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.708650 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.709074 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33126: BlockingRecv error: recv got EOF from 127.0.0.1:33126 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:58:10.786638 23673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47583cb918ad4176a43cbcafd7231150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47583cb918ad4176a43cbcafd7231150" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 40513 } } peers { permanent_uuid: "f987e0c6e8374ea4a5ffdb6b8e006e35" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 46301 } } peers { permanent_uuid: "78a83133791f4badbbb1514ec4841542" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 42183 } } }
I20260927 03:58:10.786855 23673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47583cb918ad4176a43cbcafd7231150 [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:10.967437 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:10.966656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.1.50.125:46301)
0927 03:58:10.966743 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:10.966747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:10.966758 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:10.966813 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:10.966814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:10.966829 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:10.966884 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:10.967344 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33138: BlockingRecv error: recv got EOF from 127.0.0.1:33138 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:58:11.339924 23651 catalog_manager.cc:1259] Loaded cluster ID: 0a7c29ff42df40758a848548aea85703
I20260927 03:58:11.340003 23651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: loading cluster ID for follower catalog manager: success
I20260927 03:58:11.340817 23651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: acquiring CA information for follower catalog manager: success
I20260927 03:58:11.341096 23651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78a83133791f4badbbb1514ec4841542: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:58:11.407403 23694 catalog_manager.cc:1259] Loaded cluster ID: 0a7c29ff42df40758a848548aea85703
I20260927 03:58:11.407507 23694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: loading cluster ID for follower catalog manager: success
I20260927 03:58:11.408499 23694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: acquiring CA information for follower catalog manager: success
I20260927 03:58:11.408870 23694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f987e0c6e8374ea4a5ffdb6b8e006e35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:58:11.483909 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:11.480924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.1.50.125:46301)
0927 03:58:11.481117 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:11.481119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:11.481204 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:58:11.481265 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:11.481266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:11.481322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:11.481415 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:11.481616 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.482172 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.483042 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.483247 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.483303 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:11.483485 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:11.483628 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:11.483692 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:11.483796 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:58:11.483818 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260927 03:58:11.488567 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:11.486106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.1.50.124:42183)
0927 03:58:11.486268 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:11.486271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:11.486281 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:11.486328 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:11.486328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:11.486370 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:11.486444 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:11.486572 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.487088 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.487838 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.488050 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.488105 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:11.488241 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:11.488350 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:11.488413 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:11.488453 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 03:58:11.488476 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260927 03:58:11.494155 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:11.490925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.1.50.124:42183)
0927 03:58:11.490969 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:11.490972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:11.491130 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:58:11.491257 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:11.491259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:11.491272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:11.491303 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:11.491507 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.492280 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493374 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.493641 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493700 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:11.493870 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:11.493905 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:11.493945 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:11.494073 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:58:11.494098 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:11.494131 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:11.491137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.1.50.125:46301)
0927 03:58:11.491217 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:11.491220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:11.491265 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:58:11.491351 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:11.491352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:11.491364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:11.491428 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:11.491574 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.492111 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493272 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.493545 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493589 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:11.493783 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:11.493824 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:11.493849 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:11.493971 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:58:11.494046 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:58:11.494355 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:11.490889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.1.50.126:40513)
0927 03:58:11.491056 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:11.491059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:11.491068 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:11.491151 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:11.491151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:11.491207 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:11.491286 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:11.491471 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.492271 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493237 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:11.493500 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:11.493571 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:11.493824 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:11.493996 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:11.494121 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:11.494190 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:58:11.494230 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260927 03:58:11.496950 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23416
I20260927 03:58:11.502924 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23491
I20260927 03:58:11.508553 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23566
2026-09-27T03:58:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2715 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:11 dist-test-slave-96fg krb5kdc[23782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:11 dist-test-slave-96fg krb5kdc[23782](info): set up 2 sockets
Sep 27 03:58:11 dist-test-slave-96fg krb5kdc[23782](info): commencing operation
krb5kdc: starting...
W20260927 03:58:12.482210 1225 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/test-user.keytab.
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.50.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
Entry for principal kudu/127.1.50.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab.
I20260927 03:58:12.601126 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39859
--webserver_interface=127.1.50.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:12.681908 23795 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:12.682210 23795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:12.682237 23795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:12.682266 23795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:12.683494 23795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:12.683542 23795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:12.683559 23795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:12.683576 23795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:12.683591 23795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:12.685130 23795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.126.keytab
--principal=kudu/127.1.50.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.126:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.50.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:12.685384 23795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:12.685593 23795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:12.690719 23800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:12.690701 23803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:12.690704 23801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:12.691262 23795 server_base.cc:1034] running on GCE node
I20260927 03:58:12.691574 23795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:12.691905 23795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:12.693092 23795 hybrid_clock.cc:648] HybridClock initialized: now 1790481492693078 us; error 34 us; skew 500 ppm
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:12.695907 23795 init.cc:377] Logged in from keytab as kudu/127.1.50.126@KRBTEST.COM (short username kudu)
I20260927 03:58:12.696308 23795 webserver.cc:460] Webserver started at http://127.1.50.126:45063/ using document root <none> and password file <none>
I20260927 03:58:12.696527 23795 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:12.696574 23795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:12.696694 23795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:12.697747 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data/instance:
uuid: "abe84ad06d064b16974f6eed3f94c8fa"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "86f587040602b32648184d3154d5cafa"
server_key_iv: "e950aad274f0665b348bbd7b966c4daa"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.698135 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance:
uuid: "abe84ad06d064b16974f6eed3f94c8fa"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "86f587040602b32648184d3154d5cafa"
server_key_iv: "e950aad274f0665b348bbd7b966c4daa"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.699514 23795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:12.700275 23810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:12.700528 23795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:12.700604 23795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal
uuid: "abe84ad06d064b16974f6eed3f94c8fa"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "86f587040602b32648184d3154d5cafa"
server_key_iv: "e950aad274f0665b348bbd7b966c4daa"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.700676 23795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:12.712844 23795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:12.715737 23795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:12.716028 23795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:12.720700 23862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.126:39859 every 8 connection(s)
I20260927 03:58:12.720700 23795 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.126:39859
I20260927 03:58:12.721283 23795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:12.722426 23863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:12.724953 23863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:12.726039 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23795
I20260927 03:58:12.726136 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-0/wal/instance
I20260927 03:58:12.726337 1225 external_mini_cluster.cc:1428] Setting key acdfad2e2c28990c6232671b7effe0d0
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.50.125@KRBTEST.COM; defaulting to no policy
I20260927 03:58:12.735329 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.725681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39859 (local address 127.0.0.1:42208)
0927 03:58:12.725901 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.725951 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:12.726638 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:12.726953 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:12.726956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:12.727287 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:12.727517 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.727523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.728568 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.728568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.729230 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.729232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.729755 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.729756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.729826 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.730436 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:12.730445 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:12.731989 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:12.733782 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.733789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.733791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.734047 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.734287 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.734288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.734288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.734363 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.734747 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:12.734754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:12.734826 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 03:58:12.734913 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:58:12.735078 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260927 03:58:12.735339 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.725875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.1.50.126:39859)
0927 03:58:12.726043 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:12.726048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:12.726084 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:58:12.726818 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:12.726820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:12.726839 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:12.726925 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:12.727618 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.728452 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.729360 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.729663 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.729761 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.732086 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:12.732094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:12.732094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:12.732100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:12.733708 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.734119 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.734122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.734123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.734229 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.734468 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.734468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.734468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.734674 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:12.734735 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:12.734996 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:58:12.735171 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.125@KRBTEST.COM" created.
I20260927 03:58:12.736038 23863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:12.736404 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.736288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42564)
0927 03:58:12.736338 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.736383 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:58:12.736539 23813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
I20260927 03:58:12.736967 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.736857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42580)
0927 03:58:12.736918 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.736947 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 03:58:12.737079 23863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:44425: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
Entry for principal kudu/127.1.50.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab.
I20260927 03:58:12.754460 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:44425
--webserver_interface=127.1.50.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:12.786316 23863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } attempt: 1
I20260927 03:58:12.786808 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.786656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42588)
0927 03:58:12.786718 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.786769 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:58:12.787235 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.787127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42590)
0927 03:58:12.787178 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.787209 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:58:12.787344 23863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:44425: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
W20260927 03:58:12.838207 23870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:12.838455 23870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:12.838481 23870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:12.838510 23870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:12.839766 23870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:12.839814 23870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:12.839828 23870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:12.839839 23870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:12.839850 23870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:12.841410 23870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.125.keytab
--principal=kudu/127.1.50.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.125:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.50.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:12.841655 23870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:12.841869 23870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:12.846629 23875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:12.846637 23876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:12.846664 23878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:12.846892 23870 server_base.cc:1034] running on GCE node
I20260927 03:58:12.847102 23870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:12.847384 23870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:12.848538 23870 hybrid_clock.cc:648] HybridClock initialized: now 1790481492848522 us; error 29 us; skew 500 ppm
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:12.851080 23870 init.cc:377] Logged in from keytab as kudu/127.1.50.125@KRBTEST.COM (short username kudu)
I20260927 03:58:12.851564 23870 webserver.cc:460] Webserver started at http://127.1.50.125:44289/ using document root <none> and password file <none>
I20260927 03:58:12.851778 23870 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:12.851826 23870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:12.851943 23870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:12.852991 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data/instance:
uuid: "cb6ded936b184f28900407f6c3ef8fa6"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "2e9b0b29d3e48dfc4112ac6cff6a93c3"
server_key_iv: "8103bfe1bf653b6379bc985146a61c76"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.853307 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal/instance:
uuid: "cb6ded936b184f28900407f6c3ef8fa6"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "2e9b0b29d3e48dfc4112ac6cff6a93c3"
server_key_iv: "8103bfe1bf653b6379bc985146a61c76"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.854635 23870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:12.855470 23885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:12.855501 23863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } attempt: 2
I20260927 03:58:12.855845 23870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:12.855908 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.855770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42600)
0927 03:58:12.855837 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.855877 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:58:12.855942 23870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal
uuid: "cb6ded936b184f28900407f6c3ef8fa6"
format_stamp: "Formatted at 2026-09-27 03:58:12 on dist-test-slave-96fg"
server_key: "2e9b0b29d3e48dfc4112ac6cff6a93c3"
server_key_iv: "8103bfe1bf653b6379bc985146a61c76"
server_key_version: "encryptionkey@0"
I20260927 03:58:12.856019 23870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:12.856235 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.856160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42608)
0927 03:58:12.856188 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.856212 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 03:58:12.856357 23863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.125:44425: Network error: Client connection negotiation failed: client connection to 127.1.50.125:44425: connect: Connection refused (error 111)
I20260927 03:58:12.872287 23870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:12.874250 23870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:12.874394 23870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:12.878123 23870 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.125:44425
I20260927 03:58:12.878170 23937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.125:44425 every 8 connection(s)
I20260927 03:58:12.878501 23870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/data/info.pb
I20260927 03:58:12.879184 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23870
I20260927 03:58:12.879303 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-1/wal/instance
I20260927 03:58:12.879475 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:12.879501 1225 external_mini_cluster.cc:1428] Setting key 04b12103f9cea7d66b388646d540b9e9
I20260927 03:58:12.880966 23938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.50.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.50.124@KRBTEST.COM" created.
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:58:12.893965 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.881630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.1.50.126:39859)
0927 03:58:12.881694 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:12.881697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:12.881919 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:58:12.882522 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:12.882523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:12.882535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:12.882604 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:12.883422 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.883933 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.885390 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.885596 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.885654 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.891019 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:12.891021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:12.891022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:12.891031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:12.892414 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.892925 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.892926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.892927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.892986 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.893236 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.893237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.893237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.893380 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:12.893441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:12.893763 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:58:12.893867 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:58:12.894167 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.881601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39859 (local address 127.0.0.1:42224)
0927 03:58:12.881767 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.881814 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:12.882423 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:12.882672 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:12.882676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:12.883024 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:12.883299 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.883305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.884024 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.884025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.885294 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.885298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.885697 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.885698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.885858 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.886403 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:12.886413 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:12.890881 (+ 4468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:12.892526 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.892537 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.892539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.892810 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.893051 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.893052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.893052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.893174 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.893450 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:12.893459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:12.893634 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:58:12.893733 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:58:12.893910 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 03:58:12.894784 23938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:12.903412 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.895118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42616)
0927 03:58:12.895169 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.895180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:12.895225 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:12.895529 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:12.895530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:12.895646 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:12.895704 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.895705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.897007 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.897009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.897571 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.897573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.898166 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.898167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.898228 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.898674 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:12.898678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:12.900699 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:12.902456 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.902457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.902458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.902609 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.902779 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.902780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.902780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.902832 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.903085 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:12.903086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:12.903123 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:58:12.903178 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:58:12.903285 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:58:12.903504 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.895173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.1.50.125:44425)
0927 03:58:12.895358 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:12.895363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:12.895395 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:58:12.895418 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:12.895420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:12.895435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:12.895514 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:12.895775 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.896897 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.897657 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.898095 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.898156 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.900792 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:12.900801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:12.900802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:12.900808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:12.902381 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.902678 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.902682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.902682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.902729 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.902880 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.902881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.902881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.903029 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:12.903097 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:12.903216 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:58:12.903358 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260927 03:58:12.904249 23938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:12.904671 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.904557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40186)
0927 03:58:12.904594 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.904646 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:58:12.904850 23887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
I20260927 03:58:12.905295 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.905223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40202)
0927 03:58:12.905252 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.905277 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 03:58:12.905469 23938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:38829: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
Entry for principal kudu/127.1.50.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab.
I20260927 03:58:12.907078 1225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBdoVgk/build/release/bin/kudu
/tmp/dist-test-taskBdoVgk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:38829
--webserver_interface=127.1.50.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.50.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:12.947549 23863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } attempt: 3
Sep 27 03:58:12 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:12.954694 23938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } attempt: 1
I20260927 03:58:12.955173 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.955056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40218)
0927 03:58:12.955115 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.955143 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:58:12.955529 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.947993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42630)
0927 03:58:12.948067 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.948080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:12.948146 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:12.948277 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:12.948277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:12.948441 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:12.948507 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.948509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.949497 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.949498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.950133 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:12.950137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.950615 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.950616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:12.950726 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.951198 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:12.951203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:12.952467 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:12.954203 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.954204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.954204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.954412 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.954802 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:12.954803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:12.954804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:12.954915 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:12.955227 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:12.955228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:12.955299 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 03:58:12.955373 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:12.955465 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 03:58:12.955606 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.948083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.1.50.125:44425)
0927 03:58:12.948149 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:12.948152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:12.948164 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:12.948193 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:12.948194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:12.948208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:12.948267 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:12.948581 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.949363 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.950211 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:12.950529 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:12.950601 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:12.952553 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:12.952555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:12.952556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:12.952565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:12.954105 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.954582 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.954584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.954584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.954694 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:12.954989 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:12.954990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:12.954991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:12.955149 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:12.955227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:12.955417 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:58:12.955513 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:58:12.955942 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.955853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40228)
0927 03:58:12.955887 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.955920 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 03:58:12.956070 23938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:38829: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
I20260927 03:58:12.956135 23863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:12.956478 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.956367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40234)
0927 03:58:12.956419 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.956452 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:58:12.956777 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.956696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40238)
0927 03:58:12.956733 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.956759 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:58:12.956859 23863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:38829: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
I20260927 03:58:12.988026 23863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } attempt: 1
I20260927 03:58:12.988587 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.988429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40240)
0927 03:58:12.988503 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.988549 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:58:12.988933 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:12.988837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40246)
0927 03:58:12.988884 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:12.988912 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:58:12.989027 23863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.50.124:38829: Network error: Client connection negotiation failed: client connection to 127.1.50.124:38829: connect: Connection refused (error 111)
W20260927 03:58:12.990502 23945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:58:12.990690 23945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:12.990706 23945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:12.990725 23945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:12.991811 23945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:12.991843 23945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:12.991855 23945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:12.991866 23945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:12.991882 23945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:12.993259 23945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.50.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.50.126:39859,127.1.50.125:44425,127.1.50.124:38829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/krb5kdc/kudu_127.1.50.124.keytab
--principal=kudu/127.1.50.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.50.124:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.50.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:12.993448 23945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:12.993616 23945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:12.998190 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:12.998277 23945 server_base.cc:1034] running on GCE node
W20260927 03:58:12.998241 23950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:12.998205 23953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:12.998648 23945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:12.998955 23945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:13.000104 23945 hybrid_clock.cc:648] HybridClock initialized: now 1790481493000087 us; error 34 us; skew 500 ppm
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:13.002607 23945 init.cc:377] Logged in from keytab as kudu/127.1.50.124@KRBTEST.COM (short username kudu)
I20260927 03:58:13.003038 23945 webserver.cc:460] Webserver started at http://127.1.50.124:44827/ using document root <none> and password file <none>
I20260927 03:58:13.003255 23945 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:13.003304 23945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:13.003437 23945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:13.004490 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data/instance:
uuid: "c9cfe4276f6742e5afd28708aafd019a"
format_stamp: "Formatted at 2026-09-27 03:58:13 on dist-test-slave-96fg"
server_key: "c14119cd49b93f392959e179710ab68b"
server_key_iv: "20b3a86e4561024bd40e74fd9596a807"
server_key_version: "encryptionkey@0"
I20260927 03:58:13.004842 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal/instance:
uuid: "c9cfe4276f6742e5afd28708aafd019a"
format_stamp: "Formatted at 2026-09-27 03:58:13 on dist-test-slave-96fg"
server_key: "c14119cd49b93f392959e179710ab68b"
server_key_iv: "20b3a86e4561024bd40e74fd9596a807"
server_key_version: "encryptionkey@0"
I20260927 03:58:13.006189 23945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:13.006940 23960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:13.007097 23945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:13.007179 23945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data,/tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal
uuid: "c9cfe4276f6742e5afd28708aafd019a"
format_stamp: "Formatted at 2026-09-27 03:58:13 on dist-test-slave-96fg"
server_key: "c14119cd49b93f392959e179710ab68b"
server_key_iv: "20b3a86e4561024bd40e74fd9596a807"
server_key_version: "encryptionkey@0"
I20260927 03:58:13.007246 23945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:13.016525 23945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:13.018610 23945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:13.018749 23945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:13.022900 23945 rpc_server.cc:307] RPC server started. Bound to: 127.1.50.124:38829
I20260927 03:58:13.022964 24012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.50.124:38829 every 8 connection(s)
I20260927 03:58:13.023306 23945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/data/info.pb
I20260927 03:58:13.024341 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:13.024230 23938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } attempt: 2
I20260927 03:58:13.026780 24013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.125@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:13.031926 1225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBdoVgk/build/release/bin/kudu as pid 23945
I20260927 03:58:13.032032 1225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBdoVgk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481222547165-1225-0/minicluster-data/master-2/wal/instance
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
I20260927 03:58:13.032225 1225 external_mini_cluster.cc:1428] Setting key eb6b33e7639315130373cb535b209ca1
I20260927 03:58:13.034327 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.024636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40256)
0927 03:58:13.024702 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:13.024718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:13.024784 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:13.025725 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:13.025727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:13.025887 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:13.025936 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.025938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.027476 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.027478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.028274 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.028278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.029143 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.029144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.029195 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.029680 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:13.029685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:13.031109 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:13.033012 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.033013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.033014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.033219 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.033487 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.033488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.033488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.033550 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.033982 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:13.033983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:13.034058 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 03:58:13.034135 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:58:13.034223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:58:13.034773 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.024804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.1.50.124:38829)
0927 03:58:13.024985 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.024995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.025028 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:58:13.025473 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.025483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.025548 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.025713 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.026074 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.027371 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.028422 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.029060 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.029169 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.031187 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.031195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.031195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.031202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.032939 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.033298 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.033302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.033302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.033411 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.033611 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.033612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.033612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.033896 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.033984 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.034421 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.034556 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260927 03:58:13.036479 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.027525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.1.50.126:39859)
0927 03:58:13.027587 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.027589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.027747 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:58:13.027847 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.027848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.027857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.027913 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.028449 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.029147 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.030207 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.030463 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.030530 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.033394 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.033395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.033396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.033404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.035396 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.035720 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.035721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.035721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.035778 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.035963 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.035964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.035964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.036084 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.036143 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.036318 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.036397 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:58:13.036505 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.027480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.126:39859 (local address 127.0.0.1:42232)
0927 03:58:13.027645 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:13.027689 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:13.027766 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:13.027943 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:13.027946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:13.028289 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:13.028378 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.028381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.029235 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.029236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.030132 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.030135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.030566 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.030567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.030619 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.031062 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:13.031070 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:13.033315 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:13.035466 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.035470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.035472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.035657 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.035843 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.035844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.035845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.035904 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.036139 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:13.036146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:13.036198 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 03:58:13.036284 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:13.036384 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260927 03:58:13.036675 23938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: Bootstrap starting.
I20260927 03:58:13.037001 24013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:13.037547 23938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:13.037840 23938 log.cc:824] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:13.038983 23938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: No bootstrap required, opened a new log
I20260927 03:58:13.040961 23938 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.041105 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.032563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.1.50.124:38829)
0927 03:58:13.032703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.032704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.032773 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:58:13.032831 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.032832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.032845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.032943 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.033508 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.034215 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.035292 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.035575 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.035634 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.037991 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.037993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.037993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.037999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.040094 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.040391 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.040392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.040392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.040452 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.040610 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.040610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.040611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.040737 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.040875 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.040912 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.040986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260927 03:58:13.041106 23938 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6ded936b184f28900407f6c3ef8fa6, State: Initialized, Role: FOLLOWER
I20260927 03:58:13.041183 23938 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.041509 23938 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:13.041555 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.041643 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:58:13.043355 24031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:13.043411 24031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:13.043676 23870 master_runner.cc:186] Error getting master registration for 127.1.50.126:39859: OK, instance_id { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" instance_seqno: 1790481492719056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:58:13.046533 23870 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.126@KRBTEST.COM
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
I20260927 03:58:13.050145 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.041732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.1.50.126:39859)
0927 03:58:13.041773 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.041775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.041825 (+ 50us) server_negotiation.cc:407] Connection header received
0927 03:58:13.041910 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.041910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.041921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.041996 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.042517 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.043137 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.044221 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.044493 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.044530 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.047497 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.047499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.047499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.047506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.048963 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049312 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.049313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.049314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.049378 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049569 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.049570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.049570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.049688 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.049828 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.049941 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.050052 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:13.050274 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.038216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.125:44425 (local address 127.0.0.1:42646)
0927 03:58:13.042728 (+ 4512us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:13.042738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:13.042835 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:13.043046 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:13.043047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:13.043184 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:13.043238 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.043240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.044056 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.044057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.044791 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.044793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.045248 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.045249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.045319 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.045646 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:13.045651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:13.047341 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:13.049187 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.049189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.049373 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.049575 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.049577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.049636 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.049953 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:13.049954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:13.049997 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:58:13.050095 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:58:13.050182 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4496}
I20260927 03:58:13.050616 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.038469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.1.50.125:44425)
0927 03:58:13.038519 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.038521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.042784 (+ 4263us) server_negotiation.cc:407] Connection header received
0927 03:58:13.042946 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.042947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.042957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.042986 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.043325 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.043966 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.044864 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.045136 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.045197 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.047446 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.047447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.047448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.047455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.049105 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049440 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.049441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.049441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.049499 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.049704 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.049705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.049705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.049860 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.049924 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.050460 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.050538 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:58:13.050956 24013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.50.125@KRBTEST.COM
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.124@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:13.058615 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.051210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.1.50.124:38829)
0927 03:58:13.051244 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.051246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.051277 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:58:13.051337 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.051338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.051346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.051387 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.051594 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.052360 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.053330 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.053609 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.053671 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.055948 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.055949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.055949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.055955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.057271 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057549 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.057550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.057550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.057599 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057774 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.057775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.057775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.057933 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.058041 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.058184 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.058270 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:13.058691 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.050829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.1.50.125:44425)
0927 03:58:13.050868 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.050870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.050957 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:58:13.051080 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.051080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.051090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.051118 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.051706 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.052378 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.053007 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.053198 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.053251 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.055545 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.055547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.055547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.055554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.057068 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057670 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.057671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.057672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.057719 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057912 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.057913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.057914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.058071 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.058235 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.058271 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.058343 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:13.058930 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.051180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40280)
0927 03:58:13.051241 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:13.051249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:13.051290 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:13.051388 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:13.051388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:13.051507 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:13.051536 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.051537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.052447 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.052449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.053229 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.053232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.053678 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.053679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.053732 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.054133 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:13.054137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:13.055851 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:13.057329 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.057330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.057487 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.057650 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.057651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.057651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.057705 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.058015 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:13.058040 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:13.058083 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:58:13.058761 (+ 678us) client_negotiation.cc:240] Negotiation successful
0927 03:58:13.058862 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:58:13.060578 1225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:13.060683 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: Bootstrap starting.
I20260927 03:58:13.061244 24013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:13.061453 24013 log.cc:824] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:13.062296 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: No bootstrap required, opened a new log
I20260927 03:58:13.061694 23837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "abe84ad06d064b16974f6eed3f94c8fa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42236
I20260927 03:58:13.064175 23863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } attempt: 2
I20260927 03:58:13.064281 24013 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.064445 24013 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cfe4276f6742e5afd28708aafd019a, State: Initialized, Role: FOLLOWER
I20260927 03:58:13.064536 24013 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.064877 24013 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:13.064841 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.065116 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:58:13.067345 24045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:13.067435 24045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:13.067317 23945 master_runner.cc:186] Error getting master registration for 127.1.50.126:39859: OK, instance_id { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" instance_seqno: 1790481492719056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 27 03:58:13 dist-test-slave-96fg krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.50.126@KRBTEST.COM for kudu/127.1.50.124@KRBTEST.COM
I20260927 03:58:13.069306 23945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:58:13.072436 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.064469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.50.124:38829 (local address 127.0.0.1:40284)
0927 03:58:13.064525 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:13.064536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:13.064574 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:13.064796 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:13.064797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:13.064998 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:13.065053 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.065055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.066054 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.066055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.066863 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:13.066866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.067534 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.067535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:13.067598 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.068091 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:13.068097 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:13.069585 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:13.071448 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.071449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.071450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.071615 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.071793 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:13.071808 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:13.071809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:13.071866 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:13.072103 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:13.072104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:13.072163 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 03:58:13.072224 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:13.072333 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:58:13.073022 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.064646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.1.50.124:38829)
0927 03:58:13.064685 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.064687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.064695 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:13.064716 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.064716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.064725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.065124 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.065155 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.065886 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.067053 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.067445 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.067780 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.069675 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.069676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.069677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.069684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.071338 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.071682 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.071683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.071684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.071740 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.071911 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.071912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.071912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.072056 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.072113 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.072846 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.072931 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:13.074605 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: Bootstrap starting.
I20260927 03:58:13.075309 23863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:13.075613 23863 log.cc:824] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:13.076530 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: No bootstrap required, opened a new log
I20260927 03:58:13.077868 23863 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.078006 23863 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe84ad06d064b16974f6eed3f94c8fa, State: Initialized, Role: FOLLOWER
I20260927 03:58:13.078158 23863 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.078503 23863 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:13.078516 23837 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:58:13.078606 23837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:13.079401 23837 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.079648 23837 leader_election.cc:288] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [CANDIDATE]: Term 1 election: Requested vote from peers cb6ded936b184f28900407f6c3ef8fa6 (127.1.50.125:44425), c9cfe4276f6742e5afd28708aafd019a (127.1.50.124:38829)
I20260927 03:58:13.080196 23987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abe84ad06d064b16974f6eed3f94c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9cfe4276f6742e5afd28708aafd019a"
I20260927 03:58:13.080334 23987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:13.080363 23912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abe84ad06d064b16974f6eed3f94c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6ded936b184f28900407f6c3ef8fa6"
I20260927 03:58:13.080466 23912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:13.081310 23912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe84ad06d064b16974f6eed3f94c8fa in term 1.
I20260927 03:58:13.081393 23987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe84ad06d064b16974f6eed3f94c8fa in term 1.
I20260927 03:58:13.082211 23813 leader_election.cc:302] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abe84ad06d064b16974f6eed3f94c8fa, cb6ded936b184f28900407f6c3ef8fa6; no voters:
I20260927 03:58:13.078469 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.082934 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:13.083014 24048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:13.083073 24048 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [term 1 LEADER]: Becoming Leader. State: Replica: abe84ad06d064b16974f6eed3f94c8fa, State: Running, Role: LEADER
I20260927 03:58:13.083173 24048 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } }
I20260927 03:58:13.083599 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe84ad06d064b16974f6eed3f94c8fa. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.083698 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:13.084329 23795 master_runner.cc:263] No leader master found from master abe84ad06d064b16974f6eed3f94c8fa
I20260927 03:58:13.084375 23795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:58:13.090588 24059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:13.090668 24059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:13.094592 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.082476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.1.50.126:39859)
0927 03:58:13.082514 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.082517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.086531 (+ 4014us) server_negotiation.cc:407] Connection header received
0927 03:58:13.086650 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.086651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.086663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.086694 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.087473 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.088224 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.089268 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.089528 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.089575 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.091884 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.091886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.091886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.091894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.093314 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.093702 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.093703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.093704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.093757 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.093952 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.093953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.093954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.094121 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.094191 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.094416 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.094498 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:13.095252 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.082975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.1.50.125:44425)
0927 03:58:13.083012 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.083014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.086533 (+ 3519us) server_negotiation.cc:407] Connection header received
0927 03:58:13.086672 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.086673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.086685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.086713 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.087516 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.088224 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.090130 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.090455 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.090511 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.091971 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.091972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.091973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.091980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.093796 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.094168 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.094169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.094169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.094224 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.094410 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.094411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.094412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.094531 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.094607 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.095105 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.095181 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:13.095575 24068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:13.095762 24068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:13.097240 23987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [term 1 FOLLOWER]: Refusing update from remote peer abe84ad06d064b16974f6eed3f94c8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:13.097239 23912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [term 1 FOLLOWER]: Refusing update from remote peer abe84ad06d064b16974f6eed3f94c8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:13.097585 24048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:13.097747 24064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:13.098943 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe84ad06d064b16974f6eed3f94c8fa. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.099028 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:13.099092 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe84ad06d064b16974f6eed3f94c8fa. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.099161 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:13.099970 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.100047 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:13.100065 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.100135 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:13.100337 24071 mvcc.cc:205] Tried to move back new op lower bound from 7333812195723980800 to 7333812195669491712. Current Snapshot: MvccSnapshot[applied={T|T < 7333812195723980800}]
I20260927 03:58:13.100704 24072 mvcc.cc:205] Tried to move back new op lower bound from 7333812195723980800 to 7333812195669491712. Current Snapshot: MvccSnapshot[applied={T|T < 7333812195723980800}]
I20260927 03:58:13.101572 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.101645 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:13.101868 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abe84ad06d064b16974f6eed3f94c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe84ad06d064b16974f6eed3f94c8fa" member_type: VOTER last_known_addr { host: "127.1.50.126" port: 39859 } } peers { permanent_uuid: "cb6ded936b184f28900407f6c3ef8fa6" member_type: VOTER last_known_addr { host: "127.1.50.125" port: 44425 } } peers { permanent_uuid: "c9cfe4276f6742e5afd28708aafd019a" member_type: VOTER last_known_addr { host: "127.1.50.124" port: 38829 } } }
I20260927 03:58:13.101941 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:13.102459 24068 catalog_manager.cc:1347] Generated new cluster ID: 02b17304d72b4517a93d3a2efbc4982f
I20260927 03:58:13.102509 24068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:13.110395 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:13.086286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.1.50.124:38829)
0927 03:58:13.086325 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.086327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.094198 (+ 7871us) server_negotiation.cc:407] Connection header received
0927 03:58:13.094307 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.094309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.094320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.094348 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.096555 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.097313 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.105209 (+ 7896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:13.105639 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:13.105689 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:13.107355 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:13.107356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:13.107357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:13.107365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:13.109045 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.109498 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.109499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.109500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.109562 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:13.109768 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:13.109768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:13.109769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:13.109908 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:13.109987 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:13.110211 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:58:13.110295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:13.113003 24068 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:13.113798 24068 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:13.121507 24068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe84ad06d064b16974f6eed3f94c8fa: Generated new TSK 0
I20260927 03:58:13.121757 24068 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:58:13.150300 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.148492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.1.50.125:44425)
0927 03:58:13.148539 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.148541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.149578 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 03:58:13.149684 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.149685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.149696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.149726 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.150224 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42674: BlockingRecv error: recv got EOF from 127.0.0.1:42674 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:58:13.151067 24018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.148918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.1.50.124:38829)
0927 03:58:13.148960 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.148962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.150561 (+ 1599us) server_negotiation.cc:407] Connection header received
0927 03:58:13.150647 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.150648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.150659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.150687 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.151006 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40302: BlockingRecv error: recv got EOF from 127.0.0.1:40302 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:58:13.151829 23866 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.149279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.1.50.126:39859)
0927 03:58:13.149317 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.149319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.151299 (+ 1980us) server_negotiation.cc:407] Connection header received
0927 03:58:13.151411 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.151412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.151424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.151453 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.151766 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42258: BlockingRecv error: recv got EOF from 127.0.0.1:42258 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:58:13.154229 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.153045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.1.50.125:44425)
0927 03:58:13.153083 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.153084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.153698 (+ 614us) server_negotiation.cc:407] Connection header received
0927 03:58:13.153801 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.153802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.153821 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.153847 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.154181 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42684: BlockingRecv error: recv got EOF from 127.0.0.1:42684 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:58:13.157132 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.156395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.1.50.125:44425)
0927 03:58:13.156428 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.156430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.156667 (+ 237us) server_negotiation.cc:407] Connection header received
0927 03:58:13.156754 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.156755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.156765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.156791 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.157081 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42690: BlockingRecv error: recv got EOF from 127.0.0.1:42690 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:58:13.161103 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.160254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.1.50.125:44425)
0927 03:58:13.160291 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.160294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.160570 (+ 276us) server_negotiation.cc:407] Connection header received
0927 03:58:13.160666 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.160667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.160678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.160706 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.161050 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42706: BlockingRecv error: recv got EOF from 127.0.0.1:42706 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:58:13.167501 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.166507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.1.50.125:44425)
0927 03:58:13.166553 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.166555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.166866 (+ 311us) server_negotiation.cc:407] Connection header received
0927 03:58:13.166977 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.166978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.166990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.167022 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.167438 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42720: BlockingRecv error: recv got EOF from 127.0.0.1:42720 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:58:13.177814 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.176888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.1.50.125:44425)
0927 03:58:13.176936 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.176938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.177212 (+ 274us) server_negotiation.cc:407] Connection header received
0927 03:58:13.177312 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.177314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.177326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.177356 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.177750 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42734: BlockingRecv error: recv got EOF from 127.0.0.1:42734 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:58:13.196414 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.195499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.1.50.125:44425)
0927 03:58:13.195542 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.195544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.195840 (+ 296us) server_negotiation.cc:407] Connection header received
0927 03:58:13.195944 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.195945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.195957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.195984 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.196354 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42740: BlockingRecv error: recv got EOF from 127.0.0.1:42740 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:58:13.230377 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.229651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.1.50.125:44425)
0927 03:58:13.229699 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.229701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.229839 (+ 138us) server_negotiation.cc:407] Connection header received
0927 03:58:13.229900 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.229901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.229914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.229963 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.230315 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42752: BlockingRecv error: recv got EOF from 127.0.0.1:42752 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:58:13.296324 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.295706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.1.50.125:44425)
0927 03:58:13.295784 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.295786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.295846 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:58:13.295899 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.295900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.295910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.295959 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.296259 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42756: BlockingRecv error: recv got EOF from 127.0.0.1:42756 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 03:58:13.426278 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.425634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.1.50.125:44425)
0927 03:58:13.425701 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.425703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.425739 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:58:13.425820 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.425822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.425835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.425888 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.426229 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42760: BlockingRecv error: recv got EOF from 127.0.0.1:42760 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 03:58:13.684316 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:13.683711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.1.50.125:44425)
0927 03:58:13.683769 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:13.683771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:13.683807 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:58:13.683832 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:13.683833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:13.683843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:13.683882 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:13.684261 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42762: BlockingRecv error: recv got EOF from 127.0.0.1:42762 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:58:14.044042 24031 catalog_manager.cc:1259] Loaded cluster ID: 02b17304d72b4517a93d3a2efbc4982f
I20260927 03:58:14.044168 24031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: loading cluster ID for follower catalog manager: success
I20260927 03:58:14.045137 24031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: acquiring CA information for follower catalog manager: success
I20260927 03:58:14.045465 24031 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb6ded936b184f28900407f6c3ef8fa6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:58:14.068150 24045 catalog_manager.cc:1259] Loaded cluster ID: 02b17304d72b4517a93d3a2efbc4982f
I20260927 03:58:14.068264 24045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: loading cluster ID for follower catalog manager: success
I20260927 03:58:14.069192 24045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: acquiring CA information for follower catalog manager: success
I20260927 03:58:14.069521 24045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9cfe4276f6742e5afd28708aafd019a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:58:14.200628 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:14.197579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.1.50.125:44425)
0927 03:58:14.197758 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:14.197761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:14.197774 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:58:14.197860 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:14.197860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:14.197916 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:14.198001 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:14.198274 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.198825 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.199690 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.199900 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.199964 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:14.200168 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:14.200338 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:14.200416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:14.200520 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:58:14.200542 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260927 03:58:14.205476 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:14.202764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.1.50.124:38829)
0927 03:58:14.202882 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:14.202884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:14.202894 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:14.202933 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:14.202934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:14.202976 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:14.203058 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:14.203233 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.203760 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.204650 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.204893 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.204969 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:14.205153 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:14.205269 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:14.205333 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:14.205370 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:58:14.205389 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260927 03:58:14.210487 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:14.207417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.1.50.126:39859)
0927 03:58:14.207547 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:14.207549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:14.207822 (+ 273us) server_negotiation.cc:407] Connection header received
0927 03:58:14.207894 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:14.207896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:14.207972 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:14.208059 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:14.208176 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.208783 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.209582 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.209806 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.209863 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:14.210070 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:14.210201 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:14.210279 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:14.210343 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:58:14.210373 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260927 03:58:14.211016 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:14.207365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.1.50.125:44425)
0927 03:58:14.207409 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:14.207411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:14.207984 (+ 573us) server_negotiation.cc:407] Connection header received
0927 03:58:14.208021 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:14.208022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:14.208033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:14.208086 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:14.208254 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.208987 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.210257 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.210516 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.210551 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:14.210727 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:14.210767 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:14.210795 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:14.210913 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:58:14.210935 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:14.211030 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:14.207754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.1.50.124:38829)
0927 03:58:14.207854 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:14.207856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:14.207866 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:14.207887 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:14.207888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:14.207898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:14.207967 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:14.208153 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.208961 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.210220 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:14.210495 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:14.210564 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:14.210734 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:14.210767 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:14.210795 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:14.210920 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 03:58:14.210957 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 03:58:14.215938 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23795
I20260927 03:58:14.222183 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23870
I20260927 03:58:14.227710 1225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBdoVgk/build/release/bin/kudu with pid 23945
2026-09-27T03:58:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2718 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5434 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271687 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST